WO2023115618A1

WO2023115618A1 - Audio/video screen projection method, device, and storage medium

Info

Publication number: WO2023115618A1
Application number: PCT/CN2021/142286
Authority: WO
Inventors: 刘兆祥
Original assignee: 深圳创维－Rgb电子有限公司
Priority date: 2021-12-22
Filing date: 2021-12-29
Publication date: 2023-06-29
Also published as: CN114302202A; CN114302202B

Abstract

The present application relates to the technical field of screen projection. Disclosed are an audio/video screen projection method and apparatus, a device, a system, and a storage medium. The method comprises: obtaining audio/video data sent by a terminal device; performing type analysis on the audio/video data on the basis of a pre-stored type database to obtain a corresponding encoding format; obtaining a target demutiplex plug-in according to the encoding format and a pre-stored plug-in registry; creating an audio/video processing thread according to the target demutiplex plug-in; and performing demutiplexing processing on the audio/video data on the basis of the audio/video processing thread to obtain an audio/video, and playing the audio/video. The present application solves the problem of poor compatibility of audio/video screen projection in the prior art, and achieves a technical effect of satisfying screen projection of different terminal devices to a screen projection device.

Description

音视频投屏方法、设备及存储介质Audio and video projection method, device and storage medium

本申请要求于2021年12月22日申请的、申请号为202111584667.7的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority to a Chinese patent application with application number 202111584667.7 filed on December 22, 2021, the entire contents of which are incorporated herein by reference.

技术领域technical field

本申请涉及投屏技术领域，尤其涉及一种音视频投屏方法、装置、设备、***及存储介质。The present application relates to the technical field of screen projection, and in particular to an audio and video screen projection method, device, equipment, system, and storage medium.

背景技术Background technique

随着网络技术的不断发展，人们对于智能产品的要求越来越高，其中，电视作为智能家居的主要产品，手机作为人们的生活必需品，将手机与电视互联，实现投屏逐渐重要。目前市场上的电视已经装载了很多投屏技术，但这些投屏技术在兼容性方面还存在一些问题，比如，同一个电视，使用不同手机进行投屏时，容易出现有画面无声音，有声音无画面等异常，这都是因为该电视无法满足多种手机投屏的音视频格式，导致兼容性较差。With the continuous development of network technology, people's requirements for smart products are getting higher and higher. Among them, TV is the main product of smart home, and mobile phones are the necessities of people's life. It is gradually important to connect mobile phones and TVs to realize screen projection. At present, TVs on the market have been loaded with many screen projection technologies, but there are still some problems in the compatibility of these screen projection technologies. Abnormalities such as no picture, this is because the TV cannot meet the audio and video formats of various mobile phone projection screens, resulting in poor compatibility.

技术问题technical problem

本申请的主要目的在于：提供一种音视频投屏方法、装置、设备、***及存储介质，旨在解决现有技术中音视频投屏存在兼容性较差的技术问题。The main purpose of this application is to provide an audio and video projection method, device, equipment, system and storage medium, aiming to solve the technical problem of poor compatibility of audio and video projection in the prior art.

技术解决方案technical solution

为实现上述目的，本申请采用如下技术方案：In order to achieve the above object, the application adopts the following technical solutions:

第一方面，本申请提供了一种音视频投屏方法，应用于投屏设备，所述方法包括：In the first aspect, the present application provides an audio and video screen projection method, which is applied to a screen projection device, and the method includes:

获取终端设备发送的音视频数据；Obtain the audio and video data sent by the terminal device;

基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；Performing type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;

根据所述编码格式和预存插件注册表，获得目标解复合插件；According to the encoding format and the pre-stored plug-in registration form, the target decompositing plug-in is obtained;

根据所述目标解复合插件，创建音视频处理线程；Create an audio and video processing thread according to the target decompounding plug-in;

基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；Decompounding the audio and video data based on the audio and video processing thread to obtain audio and video;

播放所述音频和所述视频。Play said audio and said video.

在一实施例中，上述音视频投屏方法中，所述获取终端设备发送的音视频数据的步骤之前，所述方法还包括：In one embodiment, in the above audio and video projection method, before the step of acquiring the audio and video data sent by the terminal device, the method further includes:

获取所述终端设备发送的建立连接请求；Obtain a connection establishment request sent by the terminal device;

响应所述建立连接请求，为所述终端设备分配网络协议地址；Responding to the connection establishment request, assigning a network protocol address to the terminal device;

根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道；Establishing a real-time streaming channel between the terminal device and the screen projection device according to the network protocol address;

所述获取终端设备发送的音视频数据包括：The acquisition of the audio and video data sent by the terminal device includes:

获取所述终端设备通过所述实时流传输通道发送的音视频数据。Acquiring audio and video data sent by the terminal device through the real-time streaming channel.

在一实施例中，上述音视频投屏方法中，所述基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式包括：In one embodiment, in the above audio and video projection method, performing type analysis on the audio and video data based on the pre-stored type database, and obtaining the corresponding encoding format includes:

在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式；Find the type corresponding to the audio-video data in the pre-stored type database to obtain the data format of the audio-video data;

根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。According to the data format of the audio and video data, the composite flow analysis is performed on the audio and video data to obtain a corresponding encoding format.

在一实施例中，上述音视频投屏方法中，所述在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式包括：In one embodiment, in the above audio and video projection method, the searching for the type corresponding to the audio and video data in the pre-stored type database, and obtaining the data format of the audio and video data include:

获取所述音视频数据的报文字段；Obtain the message field of the audio and video data;

将所述报文字段与预存的类型数据库中的数据类型进行匹配；Matching the message field with the data type in the pre-stored type database;

若匹配成功，则得到所述报文字段对应的类型；If the matching is successful, the type corresponding to the message field is obtained;

将所述报文字段对应的类型确定为所述音视频数据的数据格式；Determining the type corresponding to the message field as the data format of the audio and video data;

若匹配失败，则进行失败告警提示。If the matching fails, a failure alarm prompt will be given.

在一实施例中，上述音视频投屏方法中，所述根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式包括：In one embodiment, in the above audio and video screen projection method, the audio and video data is analyzed according to the data format of the audio and video data, and the corresponding encoding format obtained includes:

根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系；Obtain the field type corresponding to the data format of the audio and video data according to a preset configuration rule, wherein the preset configuration rule includes a correspondence between each data format and each field type;

根据所述字段类型，确定所述音视频数据的编码格式。Determine the encoding format of the audio and video data according to the field type.

在一实施例中，上述音视频投屏方法中，所述根据所述编码格式和预存插件注册表，获得目标解复合插件的步骤之前，所述方法还包括：In one embodiment, in the above audio and video projection method, before the step of obtaining the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry, the method further includes:

获取预存插件及其支持的数据格式；Obtain pre-stored plug-ins and their supported data formats;

建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。A mapping relationship between the data format and the pre-stored plug-in is established to obtain the pre-stored plug-in registry.

在一实施例中，上述音视频投屏方法中，所述根据所述编码格式和预存插件注册表，获得目标解复合插件包括：In one embodiment, in the above-mentioned audio and video projection method, the obtaining the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry includes:

在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。Search the pre-stored plug-in registry for a decomplexing plug-in corresponding to the encoding format to obtain a target decomposing plug-in, wherein the target decomposing plug-in is a plug-in capable of performing decomplexing processing on composite stream data in the encoding format.

在一实施例中，上述音视频投屏方法中，所述基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频包括：In one embodiment, in the above-mentioned audio and video projection method, the decomplexing processing of the audio and video data based on the audio and video processing thread, and obtaining audio and video include:

根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据；Decompounding the audio and video data according to the target decompounding plug-in to obtain audio stream data and video stream data;

对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器；Perform type analysis on the audio stream data and the video stream data respectively, and determine an audio decoder and a video decoder;

通过所述音频解码器对所述音频流数据进行解码，得到音频；Decoding the audio stream data by the audio decoder to obtain audio;

通过所述视频解码器对所述视频流数据进行解码，得到视频。The video stream data is decoded by the video decoder to obtain a video.

在一实施例中，上述音视频投屏方法中，所述对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器包括：In an embodiment, in the above-mentioned audio and video projection method, performing type analysis on the audio stream data and the video stream data respectively, and determining the audio decoder and the video decoder include:

对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；Perform type analysis on the audio stream data and the video stream data respectively to obtain the data type of the audio stream data and the data type of the video stream data;

根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；According to the data type of the audio stream data, determine the audio encoding format corresponding to the audio stream data;

根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式；According to the data type of the video stream data, determine the video encoding format corresponding to the video stream data;

根据所述音频编码格式，确定对应的音频解码器；Determine a corresponding audio decoder according to the audio encoding format;

根据所述视频编码格式，确定对应的视频解码器。According to the video coding format, a corresponding video decoder is determined.

在一实施例中，上述音视频投屏方法中，所述播放所述音频和所述视频的步骤之前，所述方法还包括：In an embodiment, in the above audio and video projection method, before the step of playing the audio and the video, the method further includes:

构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器；Construct a multimedia architecture, which includes an audio player and a video player;

所述播放所述音频和所述视频包括：Said playing said audio and said video includes:

通过所述多媒体架构同步播放所述音频和所述视频。The audio and the video are played synchronously through the multimedia architecture.

第二方面，本申请提供了一种音视频投屏装置，所述装置包括：In a second aspect, the present application provides an audio and video projection device, the device comprising:

数据获取模块，用于获取终端设备发送的音视频数据；A data acquisition module, configured to acquire audio and video data sent by the terminal device;

类型分析模块，用于基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；A type analysis module, configured to perform type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;

插件查找模块，用于根据所述编码格式和预存插件注册表，获得目标解复合插件；The plug-in search module is used to obtain the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry;

线程创建模块，用于根据所述目标解复合插件，创建音视频处理线程；A thread creation module, used to decompose plug-ins according to the target, and create an audio and video processing thread;

数据处理模块，用于基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；A data processing module, configured to perform demultiplexing processing on the audio and video data based on the audio and video processing thread to obtain audio and video;

投屏播放模块，用于播放所述音频和所述视频。A screen-casting playback module, configured to play the audio and the video.

第三方面，本申请提供了一种投屏设备，所述设备包括：In a third aspect, the present application provides a screen projection device, the device comprising:

存储器，存储有音视频投屏程序；The memory stores an audio and video projection program;

处理器，执行所述音视频投屏程序，以实现如上述的音视频投屏方法。The processor executes the audio and video screen projection program, so as to implement the above audio and video screen projection method.

第四方面，本申请提供了一种投屏***，所述***包括：In a fourth aspect, the present application provides a screen projection system, the system comprising:

终端设备；Terminal Equipment;

如上述的投屏设备；Such as the above-mentioned screen projection equipment;

所述终端设备与所述投屏设备通讯连接。The terminal device is communicatively connected with the screen projection device.

第五方面，本申请提供了一种计算机可读存储介质，所述存储介质上存储有计算机程序，所述计算机程序可被一个或多个处理器执行，以实现如上述的音视频投屏方法。In the fifth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program can be executed by one or more processors, so as to implement the above audio and video projection method .

有益效果Beneficial effect

本申请提供的上述一个或多个技术方案，可以具有如下优点或至少实现了如下技术效果：The above one or more technical solutions provided by this application may have the following advantages or at least achieve the following technical effects:

本申请提出的一种音视频投屏方法、装置、设备、***及存储介质，通过对终端设备发送的音视频数据进行类型分析，获得对应的编码格式，并根据该编码格式在预存插件注册表中查找解复合插件，获得目标解复合插件后，根据该目标解复合插件创建音视频处理线程，以对终端设备发送的音视频数据进行解复合处理，获得音频和视频，最后播放音频和视频，实现将终端设备的音视频投屏到投屏设备的目的；本申请可以针对不同类型的音视频数据对应创建音视频处理线程，可以对多种类型的音视频数据进行解复合、解码和播放等处理，兼容性更高，可以满足不同终端设备的投屏，提升了投屏设备的适用性和市场竞争力。An audio and video projection method, device, equipment, system, and storage medium proposed in this application obtain the corresponding encoding format by performing type analysis on the audio and video data sent by the terminal equipment, and store it in the pre-stored plug-in registry according to the encoding format. Find the demultiplexing plug-in in the search engine, and after obtaining the target decomposing plug-in, create an audio and video processing thread according to the target decomposing plug-in, so as to perform demultiplexing processing on the audio and video data sent by the terminal device, obtain audio and video, and finally play the audio and video. Realize the purpose of projecting the audio and video of the terminal device to the projecting device; this application can create audio and video processing threads for different types of audio and video data, and can decompound, decode and play various types of audio and video data. Processing, higher compatibility, can meet the screen projection of different terminal devices, and improve the applicability and market competitiveness of screen projection devices.

附图说明Description of drawings

为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present application. Those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

图1为本申请音视频投屏方法第一实施例的流程示意图；FIG. 1 is a schematic flow diagram of the first embodiment of the audio and video projection method of the present application;

图2为本申请涉及的投屏设备的硬件结构示意图；FIG. 2 is a schematic diagram of the hardware structure of the screen projection device involved in the present application;

图3为本申请音视频投屏方法第二实施例的流程示意图；FIG. 3 is a schematic flow diagram of the second embodiment of the audio and video projection method of the present application;

图4为图3中步骤S300的细化流程示意图；FIG. 4 is a schematic diagram of a refinement process of step S300 in FIG. 3;

图5为本申请音视频投屏装置第一实施例的功能模块示意图。FIG. 5 is a schematic diagram of functional modules of the first embodiment of the audio and video projection device of the present application.

本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。The realization, functional features and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

本发明的实施方式Embodiments of the present invention

为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例只是本申请的一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动的前提下所获得的所有其他实施例，都属于本申请保护的范围。In order to make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the application. Obviously, the described embodiments are only the present invention. Some embodiments of the application, but not all embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

需要说明，在本申请中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者***不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者***所固有的要素。在没有更多限制的情况下，由语句“包括……”限定的要素，并不排除在包括该要素的过程、方法、物品或者***中还存在另外的相同要素。另外，在本申请中，使用用于表示元件的诸如“模块”、“部件”或“单元”的后缀仅为了有利于本申请的说明，其本身没有特定的意义。因此，“模块”、“部件”或“单元”可以混合地使用。It should be noted that in this application, the term "comprising", "comprising" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or inherent to the process, method, article, or system are also included. Without further limitations, an element defined by the phrase "comprising..." does not preclude the presence of additional identical elements in the process, method, article or system comprising the element. In addition, in the present application, use of suffixes such as 'module', 'part' or 'unit' for denoting elements is only for facilitating the description of the present application and has no specific meaning by itself. Therefore, 'module', 'part' or 'unit' may be used in combination.

对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请中的具体含义。另外，各个实施例的技术方案可以相互结合，但是必须是以本领域普通技术人员能够实现为基础，当技术方案的结合出现相互矛盾或无法实现时，应当认为这种技术方案的结合不存在，也不在本申请要求的保护范围之内。Those of ordinary skill in the art can understand the specific meanings of the above terms in this application according to specific situations. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of those skilled in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist. Also not within the scope of protection required by the present application.

鉴于现有技术中音视频投屏存在兼容性较差的技术问题，本申请提供了一种音视频投屏方法，总体思路如下：In view of the technical problem of poor compatibility of audio and video screen projection in the prior art, this application provides an audio and video screen projection method, the general idea is as follows:

获取终端设备发送的音视频数据；基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；根据所述编码格式和预存插件注册表，获得目标解复合插件；根据所述目标解复合插件，创建音视频处理线程；基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；播放所述音频和所述视频。Obtain the audio and video data sent by the terminal device; perform type analysis on the audio and video data based on the pre-stored type database to obtain the corresponding encoding format; obtain the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry; according to the The target decomposite plug-in creates an audio and video processing thread; performs decompositing processing on the audio and video data based on the audio and video processing thread to obtain audio and video; and plays the audio and video.

通过上述技术方案，对终端设备发送的音视频数据进行类型分析，获得对应的编码格式，并根据该编码格式在预存插件注册表中查找解复合插件，获得目标解复合插件后，根据该目标解复合插件创建音视频处理线程，以对终端设备发送的音视频数据进行解复合处理，获得音频和视频，最后播放音频和视频，实现将终端设备的音视频投屏到投屏设备的目的；本申请可以针对不同类型的音视频数据对应创建音视频处理线程，可以对多种类型的音视频数据进行解复合、解码和播放等处理，兼容性更高，可以满足不同终端设备的投屏，提升了投屏设备的适用性和市场竞争力。Through the above technical solution, the type analysis is performed on the audio and video data sent by the terminal device to obtain the corresponding encoding format, and the decompositing plug-in is searched in the pre-stored plug-in registry according to the encoding format. The composite plug-in creates an audio and video processing thread to decompose the audio and video data sent by the terminal device, obtain audio and video, and finally play the audio and video, so as to realize the purpose of projecting the audio and video of the terminal device to the projection device; The application can create audio and video processing threads corresponding to different types of audio and video data, and can perform demultiplexing, decoding, and playback processing on various types of audio and video data. It has higher compatibility and can meet the requirements of different terminal devices. This ensures the applicability and market competitiveness of projection equipment.

实施例一Embodiment one

参照图1的流程示意图，提出本申请音视频投屏方法的第一实施例，该音视频投屏方法应用于投屏设备。所述投屏设备是指能够实现网络连接的终端设备，所述投屏设备可以是电视、会议平板、智能显示屏等电子设备。所述投屏设备可通过网络与终端设备实现通信连接，比如，终端设备和投屏设备与同一无线网络连接实现通信，其中，终端设备可以是手机、平板电脑、笔记本电脑、台式主机等电子设备。Referring to the schematic flowchart of FIG. 1 , a first embodiment of the audio and video screen projection method of the present application is proposed, and the audio and video screen projection method is applied to a screen projection device. The screen projection device refers to a terminal device capable of network connection, and the screen projection device may be electronic equipment such as a TV, a conference tablet, or a smart display. The screen projection device can communicate with the terminal device through the network, for example, the terminal device and the screen projection device are connected to the same wireless network to realize communication, wherein the terminal device can be a mobile phone, a tablet computer, a notebook computer, a desktop host and other electronic devices .

如图2所示，为投屏设备的硬件结构示意图。所述设备可以包括：处理器1001，例如CPU（Central Processing Unit，中央处理器），通信总线1002，用户接口1003，网络接口1004，存储器1005。As shown in FIG. 2 , it is a schematic diagram of a hardware structure of a screen projection device. The device may include: a processor 1001 , such as a CPU (Central Processing Unit, central processing unit), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 .

本领域技术人员可以理解，图2中示出的硬件结构并不构成对本申请投屏设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。Those skilled in the art can understand that the hardware structure shown in Figure 2 does not constitute a limitation to the screen projection device of the present application, and may include more or less components than those shown in the figure, or combine some components, or arrange different components .

具体的，通信总线1002用于实现这些组件之间的连接通信；Specifically, the communication bus 1002 is used to realize connection and communication between these components;

用户接口1003用于连接客户端，与客户端进行数据通信，用户接口1003可以包括输出单元，如显示屏、输入单元，如按键；The user interface 1003 is used to connect to the client and perform data communication with the client. The user interface 1003 may include an output unit, such as a display screen, and an input unit, such as a button;

网络接口1004用于连接后台服务器，与后台服务器进行数据通信，网络接口1004可以包括输入/输出接口，比如标准的有线接口、无线接口，如Wi-Fi接口；The network interface 1004 is used to connect to the background server and perform data communication with the background server. The network interface 1004 may include an input/output interface, such as a standard wired interface and a wireless interface, such as a Wi-Fi interface;

存储器1005用于存储各种类型的数据，这些数据例如可以包括该投屏设备中任何应用程序或方法的指令，以及应用程序相关的数据，存储器1005可以是高速RAM存储器，也可以是稳定的存储器，例如磁盘存储器，可选的，存储器1005还可以是独立于所述处理器1001的存储装置；The memory 1005 is used to store various types of data, such data may include instructions of any application or method in the screen projection device, as well as application-related data, and the memory 1005 may be a high-speed RAM memory or a stable memory , such as a disk storage, optionally, the storage 1005 may also be a storage device independent of the processor 1001;

具体的，继续参照图2，存储器1005中可以包括操作***、网络通信模块、用户接口模块以及音视频投屏程序，其中，网络通信模块主要用于连接终端设备，与终端设备进行数据通信；Specifically, continuing to refer to FIG. 2, the memory 1005 may include an operating system, a network communication module, a user interface module, and an audio and video projection program, wherein the network communication module is mainly used to connect to a terminal device and perform data communication with the terminal device;

处理器1001用于调用存储器1005中存储的音视频投屏程序，并执行以下操作：The processor 1001 is used to call the audio and video projection program stored in the memory 1005, and perform the following operations:

播放所述音频和所述视频。Play said audio and said video.

基于上述的投屏设备，下面结合图1所示的流程示意图，对本实施例的音视频投屏方法进行详细描述。所述方法可以包括以下步骤：Based on the above-mentioned screen projection device, the audio and video screen projection method of this embodiment will be described in detail below in conjunction with the schematic flowchart shown in FIG. 1 . The method may include the steps of:

步骤S110：获取终端设备发送的音视频数据。Step S110: Obtain audio and video data sent by the terminal device.

具体的，终端设备可以是手机、平板电脑、笔记本电脑和台式主机等电子设备，投屏设备可以是电视、智能显示屏、会议平板等电子设备，终端设备与投屏设备连接同一网络，实现通信连接。音视频数据是指包含音频和视频的复合流数据。投屏设备可通过实时流通道获取终端设备发送的音视频数据。Specifically, the terminal device can be electronic devices such as mobile phones, tablet computers, laptops, and desktop hosts, and the projection device can be electronic devices such as TVs, smart displays, and conference tablets. The terminal device and the projection device are connected to the same network to achieve communication. connect. Audio-video data refers to composite stream data including audio and video. The screen projection device can obtain the audio and video data sent by the terminal device through the real-time streaming channel.

步骤S120：基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式。Step S120: Perform type analysis on the audio and video data based on the pre-stored type database to obtain a corresponding encoding format.

具体的，类型是指音视频数据的复合流数据的数据格式，编码格式是指终端设备将音视频数据编码为数据包形式发送给投屏设备时采用编码器进行编码的格式。对获取到的音视频数据进行类型分析（Typefind），就是在预存的类型数据库中查找到该音视频数据的数据格式，再根据该数据格式进行编码格式的匹配，从而获得音视频数据对应的编码格式。Specifically, the type refers to the data format of the composite stream data of the audio and video data, and the encoding format refers to the format in which the terminal device encodes the audio and video data into data packets and sends them to the projection device using an encoder. Type analysis (Typefind) of the acquired audio and video data is to find the data format of the audio and video data in the pre-stored type database, and then match the encoding format according to the data format, so as to obtain the corresponding encoding of the audio and video data Format.

步骤S130：根据所述编码格式和预存插件注册表，获得目标解复合插件。Step S130: Obtain the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry.

具体的，预存插件注册表是指存储有插件与该插件对应可支持的数据格式的表格，其中，插件与数据格式存在映射关系。在投屏设备初始化时，将大量插件关联对应可支持的数据格式后，注册在一表格中，获得预存插件注册表。其中，插件可以包括解复合插件、解码插件、播放器插件等等。解复合是指将复合流数据分解为音频流和视频流。根据获得的音视频数据的编码格式，在预存插件注册表中进行查找，获得可以对该音视频数据进行解复合的插件，调取该插件，即获得目标解复合插件。Specifically, the pre-stored plug-in registry refers to a table storing a plug-in and a data format that the plug-in can support correspondingly, wherein there is a mapping relationship between the plug-in and the data format. When the screen projection device is initialized, after associating a large number of plug-ins with corresponding supported data formats, register them in a table to obtain the pre-stored plug-in registry. Wherein, the plug-in may include a demultiplexing plug-in, a decoding plug-in, a player plug-in, and the like. Decompositing refers to decomposing composite stream data into audio and video streams. According to the encoding format of the obtained audio-video data, search in the pre-stored plug-in registry to obtain a plug-in capable of decomplexing the audio-video data, and call the plug-in to obtain the target decomplexing plug-in.

步骤S140：根据所述目标解复合插件，创建音视频处理线程。Step S140: Create an audio and video processing thread according to the target decompositing plug-in.

具体的，在按照步骤S120和步骤S130确定目标解复合插件后，基于该插件创建线程（Pipeline）。线程是指具有一系列线程段的模型，每个线程段执行一个动作。本申请中的线程是在获取音视频数据后，基于该音视频数据特定的线程，具有定制性。创建音视频处理线程包括创建一系列线程段，比如，定义利用获取到的目标解复合插件对音视频数据进行解复合，得到音频流和视频流的过程对应的线程段，定义分别对音频流和视频流依次进行队列、解码器选择、解码、参数设置、播放器选择等等操作的不同过程对应的线程段。总的来说，创建音视频处理线程的目的是利用该音视频处理线程对接收到的音视频数据进行一系列操作，该操作具有重复性和连续性，可以对实时获取的音视频数据进行实时处理，从而实现终端设备上音视频的实时投屏。Specifically, after the target decompositing plug-in is determined according to step S120 and step S130, a thread (Pipeline) is created based on the plug-in. Threading refers to a model with a sequence of thread segments, each thread segment performing an action. The thread in this application is a specific thread based on the audio and video data after the audio and video data is acquired, and has customization. Creating an audio and video processing thread includes creating a series of thread segments. For example, define the thread segment corresponding to the process of decomposing the audio and video data by using the obtained target decompositing plug-in to obtain the audio stream and video stream, and define the audio stream and video stream respectively. The thread segment corresponding to the different processes of video stream sequentially performing queue, decoder selection, decoding, parameter setting, player selection and other operations. In general, the purpose of creating an audio and video processing thread is to use the audio and video processing thread to perform a series of operations on the received audio and video data. Processing, so as to realize real-time projection of audio and video on the terminal device.

步骤S150：基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频。Step S150: Based on the audio and video processing thread, perform demultiplexing processing on the audio and video data to obtain audio and video.

具体的，在步骤S140创建得到步骤S110获取的音视频数据对应的音视频处理线程后，直接利用该音视频处理线程对实时获取到的音视频数据进行处理，包括但不限于进行解复合、队列、属性分析、解码器选择、解码、参数设置等等操作，其中，上述操作的执行顺序或被执行对象可基于实际需求设定，比如，对视频流的处理可以比音频流的处理更复杂，参数设置更多等等。利用音视频处理线程对音视频数据进行解复合处理后，可以得到拆分后的音频和视频。Specifically, after the audio-video processing thread corresponding to the audio-video data obtained in step S110 is created in step S140, the audio-video processing thread is directly used to process the audio-video data obtained in real time, including but not limited to decompositing, queue , attribute analysis, decoder selection, decoding, parameter setting and other operations, wherein the execution sequence or objects to be executed of the above operations can be set based on actual needs, for example, the processing of video streams can be more complicated than the processing of audio streams, More parameter settings and more. After demultiplexing the audio and video data by using the audio and video processing thread, the split audio and video can be obtained.

步骤S160：播放所述音频和所述视频。Step S160: Play the audio and the video.

具体的，在步骤S150获得音频和视频后，投屏设备可以选择对应的音频播放器和视频播放器分别对得到的音频和视频进行播放。其中，音频播放器和视频播放器可以是投屏设备自带的音频播放器和视频播放器，也可以是基于该音视频数据独立创建的多媒体架构中的音频播放器和视频播放器。播放音频和视频时可以同步播放，以保证音画同步及实现音频输出和视频输出，即保证该终端设备的音视频可以成功在投屏设备上播放。Specifically, after the audio and video are obtained in step S150, the screen projection device may select a corresponding audio player and video player to play the obtained audio and video respectively. Wherein, the audio player and the video player may be the audio player and the video player provided by the screen projection device, or may be the audio player and the video player in the multimedia framework independently created based on the audio and video data. When playing audio and video, it can be played synchronously to ensure the synchronization of audio and video and to realize audio output and video output, that is, to ensure that the audio and video of the terminal device can be successfully played on the screen projection device.

本实施例提供的音视频投屏方法，通过对终端设备发送的音视频数据进行类型分析，获得对应的编码格式，并根据该编码格式在预存插件注册表中查找解复合插件，获得目标解复合插件后，根据该目标解复合插件创建音视频处理线程，以对终端设备发送的音视频数据进行解复合处理，获得音频和视频，最后播放音频和视频，实现将终端设备的音视频投屏到投屏设备的目的；本申请可以针对不同类型的音视频数据对应创建音视频处理线程，可以对多种类型的音视频数据进行解复合、解码和播放等处理，兼容性更高，可以满足不同终端设备的投屏，提升了投屏设备的适用性和市场竞争力。The audio and video projection method provided in this embodiment obtains the corresponding encoding format by performing type analysis on the audio and video data sent by the terminal device, and searches for the demultiplexing plug-in in the pre-stored plug-in registry according to the encoding format to obtain the target decompositing After the plug-in, create an audio and video processing thread according to the target decompositing plug-in to decompose the audio and video data sent by the terminal device, obtain audio and video, and finally play the audio and video to realize the audio and video projection of the terminal device to the The purpose of the screen projection device; this application can create audio and video processing threads corresponding to different types of audio and video data, and can perform decompositing, decoding, and playback processing on various types of audio and video data. The compatibility is higher and can meet different needs. The screen projection of terminal equipment improves the applicability and market competitiveness of screen projection equipment.

实施例二Embodiment two

基于同一申请构思，参照图3至图4，提出本申请音视频投屏方法的第二实施例，该音视频投屏方法应用于投屏设备。Based on the same application idea, referring to FIG. 3 to FIG. 4 , a second embodiment of the audio and video screen projection method of the present application is proposed, and the audio and video screen projection method is applied to a screen projection device.

下面结合图3所示的流程示意图，对本实施例的音视频投屏方法进行详细描述。所述方法可以包括以下步骤：The audio and video projection method of this embodiment will be described in detail below with reference to the schematic flowchart shown in FIG. 3 . The method may include the steps of:

步骤S210：获取终端设备发送的建立连接请求。Step S210: Obtain the connection establishment request sent by the terminal device.

具体的，终端设备与投屏设备通过网络实现通信连接，本实施例的终端设备以手机为例，投屏设备以电视为例进行说明，也就是手机与电视连接至同一WiFi网络。用户在终端设备上发起投屏请求时，首先生成一个与投屏设备建立连接的请求，并将该请求发送至投屏设备，投屏设备即获取到终端设备发送的建立连接请求。本实施例中，手机通过WiFi网络发送建立连接请求至电视，具体通过WiFi网络的底层协商WPS（Wi-Fi Protected Setup，WiFi保护设置）技术和握手协议实现身份确认和信息发送，其中，涉及的底层协商WPS技术和握手协议均为现有技术，此处不再赘述。Specifically, the terminal device and the screen projection device are connected by communication through the network. In this embodiment, the terminal device is taken as an example, and the screen projection device is described by taking a TV as an example, that is, the mobile phone and the TV are connected to the same WiFi network. When the user initiates a screen projection request on the terminal device, a request to establish a connection with the screen projection device is first generated, and the request is sent to the screen projection device, and the screen projection device obtains the connection establishment request sent by the terminal device. In this embodiment, the mobile phone sends a connection establishment request to the TV through the WiFi network, and specifically negotiates WPS (Wi-Fi Protected Setup (WiFi Protected Setup) technology and handshake protocol to realize identity confirmation and information transmission. Among them, the underlying negotiation WPS technology and handshake protocol involved are all existing technologies, and will not be repeated here.

步骤S220：响应所述建立连接请求，为所述终端设备分配网络协议地址。Step S220: Responding to the connection establishment request, assigning a network protocol address to the terminal device.

具体的，投屏设备接收到建立连接请求后，响应于该请求，为终端设备分配网络协议地址（Internet Protocol Address，IP地址），具体通过DHCP（Dynamic Host Configuration Protocol，动态主机配置协议）向终端设备动态分配IP地址和对应的配置信息，并将该IP地址和配置信息发送给终端设备进行存储和利用，利用是指后续直接发送包含该IP地址和配置信息的建立连接请求给投屏设备。本实施例中，电视在接收到手机发送的建立连接请求后，响应该请求，为手机分配一个IP地址和对应的配置信息，并发送至手机，手机接收到该IP地址和对应的配置信息后，进行保存，后续可以直接发送包含该IP地址的建立连接请求给电视，电视则不需要再重复分配IP地址，可以节省建立连接的步骤，以便后续快速建立实时流传输通道。Specifically, after receiving the connection establishment request, the screen projection device assigns an Internet protocol address (Internet Protocol Address, IP address) to the terminal device in response to the request, and specifically sends the terminal device an IP address through DHCP (Dynamic Host Configuration Protocol, Dynamic Host Configuration Protocol). The device dynamically allocates an IP address and corresponding configuration information, and sends the IP address and configuration information to the terminal device for storage and utilization. Utilization refers to the subsequent direct sending of a connection establishment request containing the IP address and configuration information to the screen projection device. In this embodiment, after receiving the connection establishment request sent by the mobile phone, the television responds to the request, assigns an IP address and corresponding configuration information to the mobile phone, and sends them to the mobile phone, and the mobile phone receives the IP address and the corresponding configuration information , save it, and then you can directly send a connection establishment request containing the IP address to the TV, and the TV does not need to repeatedly assign the IP address, which can save the steps of establishing a connection, so that the real-time streaming channel can be quickly established in the future.

步骤S230：根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道。Step S230: Establish a real-time streaming channel between the terminal device and the screen projection device according to the network protocol address.

具体的，投屏设备为终端设备分配IP地址，且终端设备接收到该IP地址后，可以再次发送一个建立传输通道的请求给投屏设备，该请求中包含该IP地址，投屏设备接收到建立传输通道请求后，响应于该请求，根据该IP地址，通过RTSP（Real Time Streaming Protocol，实时流传输协议）建立终端设备与投屏设备之间的实时流传输通道。本实施例中，手机接收到电视分配的IP地址后，再通过WiFi网络发送一个包含该IP地址的建立传输通道的请求给电视，电视响应该请求，成功建立一个手机与电视的实时流传输通道。Specifically, the screen projection device assigns an IP address to the terminal device, and after the terminal device receives the IP address, it can send a request to establish a transmission channel to the screen projection device again. The request contains the IP address, and the screen projection device receives the IP address. After the transmission channel request is established, in response to the request, a real-time streaming transmission channel between the terminal device and the screen projection device is established through RTSP (Real Time Streaming Protocol, real-time streaming protocol) according to the IP address. In this embodiment, after the mobile phone receives the IP address assigned by the TV, it sends a request for establishing a transmission channel including the IP address to the TV through the WiFi network, and the TV responds to the request and successfully establishes a real-time streaming transmission channel between the mobile phone and the TV. .

步骤S240：获取终端设备发送的音视频数据。Step S240: Obtain audio and video data sent by the terminal device.

具体的，获取终端设备通过所述实时流传输通道发送的音视频数据。在步骤S230建立好实时流传输通道后，终端设备通过该实时流传输通道发送实时的音视频数据给投屏设备，投屏设备即可获取到该音视频数据。Specifically, the audio and video data sent by the terminal device through the real-time streaming channel is acquired. After the real-time streaming channel is established in step S230, the terminal device sends real-time audio and video data to the screen projection device through the real-time stream transmission channel, and the screen projection device can obtain the audio and video data.

步骤S250：基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式。Step S250: Perform type analysis on the audio and video data based on the pre-stored type database to obtain a corresponding encoding format.

进一步地，所述步骤S250可以包括：Further, the step S250 may include:

步骤S251：在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式。Step S251: Search the type corresponding to the audio and video data in the pre-stored type database to obtain the data format of the audio and video data.

具体的，投屏设备接收到音视频数据后，在预存的类型数据库中对音视频数据进行类型查找，匹配得到音视频数据对应的类型，作为音视频数据的数据格式。Specifically, after the screen projection device receives the audio and video data, it searches the type of the audio and video data in the pre-stored type database, matches the type corresponding to the audio and video data, and uses it as the data format of the audio and video data.

具体的，步骤S251可以包括：Specifically, step S251 may include:

步骤S251.1：获取所述音视频数据的报文字段。Step S251.1: Obtain the message field of the audio and video data.

在传输音视频数据时，一般以报文的形式进行传输，在步骤S240获取到音视频数据的报文后，本步骤即可获取该报文的字段，即所述音视频数据的报文字段。When transmitting audio and video data, it is generally transmitted in the form of a message. After the message of the audio and video data is obtained in step S240, this step can obtain the field of the message, that is, the message field of the audio and video data .

步骤S251.2：将所述报文字段与预存的类型数据库中的数据类型进行匹配。Step S251.2: Match the message fields with the data types in the pre-stored type database.

预存的类型数据库为包含不同字段和不同数据类型对应关系的表格，一个字段表示一个特定的数据类型，因此，对应搜索进行匹配即可查找到该报文字段对应的数据类型。虽然预设的类型数据库可以根据实际情况预先设定很多个对应关系，但难免有某个新的数据类型不在该表格中或者字段表示出错等等情况，因此，可能有匹配成功或失败的情况。The pre-stored type database is a table containing the correspondence between different fields and different data types. A field represents a specific data type. Therefore, the corresponding data type of the message field can be found by matching the corresponding search. Although the preset type database can pre-set many corresponding relationships according to the actual situation, it is inevitable that a new data type is not in the table or the field indicates an error, etc. Therefore, the matching may succeed or fail.

步骤S251.3：若匹配成功，则得到所述报文字段对应的类型；Step S251.3: If the matching is successful, obtain the type corresponding to the message field;

步骤S251.4：将所述报文字段对应的类型确定为所述音视频数据的数据格式。Step S251.4: Determine the type corresponding to the message field as the data format of the audio and video data.

当在预存的类型数据库中查找到上述报文字段对应的数据类型时，说明投屏设备在设计之初是考虑到了该复合流数据的数据类型，对应地，一般也会针对该数据类型存储对应的解复合插件或其他插件。此处得到报文字段对应的类型后，即可将该类型确定为音视频数据的数据格式，也就是复合流数据的数据格式。When the data type corresponding to the above message field is found in the pre-stored type database, it means that the screen projection device has considered the data type of the composite stream data at the beginning of the design, and correspondingly, it will generally store the corresponding data type for this data type. decompound plugin or other plugins. Here, after obtaining the type corresponding to the message field, the type can be determined as the data format of the audio and video data, that is, the data format of the composite stream data.

步骤S251.5：若匹配失败，则进行失败告警提示。Step S251.5: If the matching fails, give a failure warning prompt.

若传输过程出错或其他原因导致获取的报文字段不完整，或者音视频数据是一种全新的复合流数据格式，不在该投屏设备设计之初考虑的复合流数据格式内，又或者还有其他缺陷或故障等等原因，都可能存在步骤S251.2匹配出错的情况，即匹配失败，此时，投屏设备可以进行失败告警提示，以提醒用户及时采取措施，比如重新建立终端设备与投屏设备的连接，重新建立实时流传输通道等等。If there is an error in the transmission process or other reasons, the obtained message field is incomplete, or the audio and video data is a new composite stream data format, which is not in the composite stream data format considered at the beginning of the design of the projection device, or there are still For other reasons such as defects or faults, there may be a matching error in step S251.2, that is, matching failure. screen device connections, re-establish live streaming channels, and more.

步骤S252：根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。Step S252: According to the data format of the audio and video data, perform composite flow analysis on the audio and video data to obtain a corresponding encoding format.

具体的，投屏设备得到音视频数据的数据格式后，由于音视频数据是复合流数据，可继续对该音视频数据进行复合流分析，查出该复合流采用的编码格式，即得到所述音视频数据对应的编码格式，比如，同一数据格式的复合流，采用的编码格式可能存在不同，因此，需要具体确定编码格式。现有的一些电视就是因为无法满足不同的编码格式，才导致后续无法播放音频或视频。Specifically, after the screen projection device obtains the data format of the audio and video data, since the audio and video data is composite stream data, it can continue to analyze the composite stream of the audio and video data, find out the encoding format used by the composite stream, and obtain the The encoding format corresponding to the audio and video data, for example, the composite stream of the same data format may adopt different encoding formats, therefore, the encoding format needs to be specifically determined. Some existing TVs cannot play audio or video later because they cannot meet different encoding formats.

具体的，步骤252可以包括：Specifically, step 252 may include:

步骤S252.1：根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系。Step S252.1: Obtain the field type corresponding to the data format of the audio and video data according to the preset configuration rule, wherein the preset configuration rule includes the corresponding relationship between each data format and each field type.

具体的，预设配置规则为数据格式-字段类型-编码格式的表格，一个数据格式可以采用多种编码格式，比如，mp4格式可以有四种编码格式，分别为H264、MPEG4、H264、VP6等等。而不同的编码格式可以利用字段类型来区分，也就是一种数据格式对应的不同编码格式，一个编码格式对应一个字段类型，也就是在预设配置规则中会有对应的字段类型表示该音视频数据的数据格式。Specifically, the preset configuration rule is a table of data format-field type-encoding format. A data format can adopt multiple encoding formats. For example, the mp4 format can have four encoding formats, namely H264, MPEG4, H264, VP6, etc. wait. Different encoding formats can be distinguished by field type, that is, different encoding formats corresponding to one data format, and one encoding format corresponds to a field type, that is, there will be a corresponding field type in the preset configuration rules to indicate the audio and video The data format of the data.

步骤S252.2：根据所述字段类型，确定所述音视频数据的编码格式。Step S252.2: Determine the encoding format of the audio and video data according to the field type.

具体的，在得到音视频数据对应的字段类型后，即可确定对应的编码格式，该编码格式即为该音视频数据所采用的编码格式。Specifically, after the field type corresponding to the audio and video data is obtained, the corresponding encoding format can be determined, and the encoding format is the encoding format adopted by the audio and video data.

步骤S260：获取预存插件及其支持的数据格式。Step S260: Obtain pre-stored plug-ins and their supported data formats.

具体的，在设计之初，初始化该方法时，可以直接存入一些常用的插件到投屏设备中，投屏设备获取到预存插件。每个插件只能对应处理预设的数据格式，因此，每个插件支持的数据格式存在不同，在获取预存插件时，同步获取该插件支持的数据格式。Specifically, at the beginning of the design, when initializing this method, some commonly used plug-ins can be directly stored in the screen projection device, and the screen projection device can obtain the pre-stored plug-ins. Each plug-in can only correspond to the preset data format. Therefore, the data format supported by each plug-in is different. When obtaining the pre-stored plug-in, the data format supported by the plug-in is obtained synchronously.

步骤S270：建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。Step S270: Establish a mapping relationship between the data format and the pre-stored plug-in, and obtain the pre-stored plug-in registry.

具体的，由于不同功能的插件可以支持的数据格式多样，有可能会存在几个插件支持同一数据格式的情况，因此，基于数据格式来设定对应的预存插件，建立数据格式与预存插件之间的映射关系。其中，预存插件可以按照不同功能进行分类，比如，A数据格式对应的预存插件有，解复合插件a1、解码插件a2等等，B数据格式对应的预存插件有，解码插件b1、加密插件b2等等，具体根据实际情况设定。设定数据格式与预存插件之间的映射关系后，将该数据格式、预存插件以及映射关系注册到一个表格中，即获得预存插件注册表，以便后续查找插件或调取插件时使用。Specifically, since plug-ins with different functions can support a variety of data formats, several plug-ins may support the same data format. Therefore, the corresponding pre-stored plug-ins are set based on the data format, and the relationship between the data format and the pre-stored plug-ins is established. mapping relationship. Among them, the pre-stored plug-ins can be classified according to different functions. For example, the pre-stored plug-ins corresponding to the A data format include decompositing plug-in a1, decoding plug-in a2, etc., and the pre-stored plug-ins corresponding to the B data format include decoding plug-in b1 and encryption plug-in b2, etc. etc., set according to the actual situation. After setting the mapping relationship between the data format and the pre-stored plug-in, register the data format, the pre-stored plug-in and the mapping relationship in a table, and obtain the pre-stored plug-in registration form for subsequent use when searching for or calling a plug-in.

步骤S280：根据所述编码格式和预存插件注册表，获得目标解复合插件。Step S280: Obtain the target demultiplexing plug-in according to the encoding format and the pre-stored plug-in registry.

具体的，所述步骤S280可以包括：Specifically, the step S280 may include:

步骤S281：在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。Step S281: Search the pre-stored plug-in registry for the decomplexing plug-in corresponding to the encoding format, and obtain the target decompositing plug-in, wherein the target decompositing plug-in is capable of performing decomplexing processing on the composite stream data in the encoding format plugin.

具体的，在步骤S250获得音视频数据对应的编码格式后，在预存插件注册表中进行查找，查找该编码格式所对应的可使用的解复合插件，也就是以该编码格式与预存插件注册表中的数据格式匹配，匹配成功时，直接获取该数据格式对应的多个插件，在这多个插件中找到解复合功能的插件，即为目标解复合插件。Specifically, after the encoding format corresponding to the audio and video data is obtained in step S250, a search is performed in the pre-stored plug-in registry to find the usable decompositing plug-in corresponding to the encoding format, that is, the encoding format and the pre-stored plug-in registry The data format matches in , and when the match is successful, multiple plug-ins corresponding to the data format are directly obtained, and the plug-in with the decomposing function is found among the multiple plug-ins, which is the target decomposing plug-in.

步骤S290：根据所述目标解复合插件，创建音视频处理线程。Step S290: Create an audio and video processing thread according to the target decompositing plug-in.

具体的，获得目标解复合插件后，创建该音视频数据专用的音视频处理线程，其中，该音视频处理线程包括多个线程段。比如，设定对该音视频数据解复合后得到的音频流进行解码的线程段，具体可以设定先对解复合得到的音频流进行编码格式的分析，得到该音频流的编码格式，再利用预存插件注册表，根据该编码格式确定该音频流的音频解码器，最后，将该音频流传送到该音频解码器中进行解码，得到音频。又比如，设定将得到的音频流传送到确定的音频解码器中进行解码的线程段，具体可以设定先对音频流进行队列排序，再根据队列依次将音频流输入到音频解码器中，输出音频，最后还可以对输出的音频进行参数设置，以获得更优质的音频等等，这里进行参数设置可以使后续利用该线程得到的音频直接调用投屏设备的本地音频播放器进行播放，而无需专门搭建播放插件。需要说明，在实际创建音视频处理线程的过程中，除了第一步是设定利用目标解复合插件对步骤S240获取的音视频数据进行解复合，得到音频流和视频流的线程段之外，之后分别对音频流和视频流进行具体处理的线程段，均可以根据实际情况设定，但音频流和视频流需保持同步，以防止后续播放出现音画不同步的问题。Specifically, after the target decompositing plug-in is obtained, an audio and video processing thread dedicated to the audio and video data is created, wherein the audio and video processing thread includes multiple thread segments. For example, to set the thread segment for decoding the audio stream obtained after demultiplexing the audio and video data, specifically, it can be set to analyze the encoding format of the audio stream obtained after decomplexing to obtain the encoding format of the audio stream, and then use The plug-in registry is pre-stored, the audio decoder of the audio stream is determined according to the encoding format, and finally, the audio stream is sent to the audio decoder for decoding to obtain audio. For another example, set the thread segment that sends the obtained audio stream to a certain audio decoder for decoding. Specifically, you can set the queue to sort the audio stream first, and then input the audio stream to the audio decoder in sequence according to the queue. Output audio, and finally you can also set the parameters of the output audio to obtain better quality audio, etc. The parameter setting here can make the audio obtained by using this thread directly call the local audio player of the screen projection device to play, and No need to build a playback plug-in specifically. It should be noted that in the process of actually creating an audio and video processing thread, except that the first step is to use the target decompositing plug-in to decompose the audio and video data obtained in step S240 to obtain the thread segments of the audio stream and video stream, Afterwards, the thread segments for specific processing of the audio stream and the video stream can be set according to the actual situation, but the audio stream and the video stream must be kept in sync to prevent the audio and video from being out of sync in subsequent playback.

在具体实施过程中，还可以在创建音视频处理线程时，将涉及的参数分离出来，由RTSP单独进行终端设备与投屏设备的交互，设置参数。比如，本次手机投屏的视频为360P，下次可能会是720P，但不论是什么参数的音视频数据，基于该终端设备的数据格式和编码格式一般不会变动，因此，可以保持创建的音视频处理线程所在的缓冲区，另外创建一个独立的缓冲区，对参数进行设置。这样，即使是重新连接的终端设备，若仅仅是参数存在变化，其他格式和IP地址均未变动的话，可以用到上次创建的音视频处理线程，节省缓存空间。创建的音视频处理线程位于投屏设备的特定缓存区，同一终端设备在后续建立连接后进行投屏时，可调用该音视频处理线程，节省投屏过程，减少延迟，防止卡顿，画面表现更流畅。In the specific implementation process, when creating the audio and video processing thread, the parameters involved can also be separated, and the RTSP can independently perform the interaction between the terminal device and the screen projection device, and set the parameters. For example, the video casted by the mobile phone is 360P this time, and it may be 720P next time. However, no matter what parameters the audio and video data are based on, the data format and encoding format of the terminal device will generally not change. Therefore, the created The buffer where the audio and video processing thread is located, and create an independent buffer to set the parameters. In this way, even for a reconnected terminal device, if only the parameters have changed and other formats and IP addresses have not changed, the audio and video processing thread created last time can be used to save cache space. The created audio and video processing thread is located in the specific buffer area of the screen projection device. When the same terminal device performs screen projection after the subsequent connection is established, the audio and video processing thread can be called to save the screen projection process, reduce delays, prevent freezes, and improve screen performance. more fluid.

步骤S300：基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频。Step S300: Perform decomplexing processing on the audio and video data based on the audio and video processing thread to obtain audio and video.

进一步地，如图4所示的流程示意图，所述步骤S300可以包括：Further, as shown in the schematic flowchart of FIG. 4, the step S300 may include:

步骤S301：根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据。Step S301: According to the target demultiplexing plug-in, decomplex the audio and video data to obtain audio stream data and video stream data.

具体的，利用音视频处理线程对音视频数据进行处理，首先利用目标解复合插件对音视频数据进行解复合，得到音频流数据和视频流数据，后续则继续根据音视频处理线程中定义的处理顺序或设定的过程分别对音视流数据和视频流数据进行处理。Specifically, the audio and video processing thread is used to process the audio and video data. First, the target decompositing plug-in is used to decompose the audio and video data to obtain audio stream data and video stream data, and then continue to process according to the definition in the audio and video processing thread. The sequential or set process processes the audio and video stream data and the video stream data respectively.

步骤S302：对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器。Step S302: Perform type analysis on the audio stream data and the video stream data respectively, and determine an audio decoder and a video decoder.

具体的，所述步骤S302可以包括：Specifically, the step S302 may include:

步骤S302.1：对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；Step S302.1: Perform type analysis on the audio stream data and the video stream data respectively to obtain the data type of the audio stream data and the data type of the video stream data;

步骤S302.2：根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；Step S302.2: Determine the audio encoding format corresponding to the audio stream data according to the data type of the audio stream data;

步骤S302.3：根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式。Step S302.3: According to the data type of the video stream data, determine the video coding format corresponding to the video stream data.

具体的，在实际实施过程中，同一数据类型的音视频数据，即某一复合流数据格式在解复合后可能会有不同格式的音频流数据以及不同格式的视频流数据，可以先确定该音频流数据的数据类型后，再获取该音频流数据的数据类型所对应的编码格式，从而得到该音频流数据对应的音频编码格式；视频流部分也一样，先确定该视频流数据的数据类型后，再获取该视频流数据的数据类型所对应的编码格式，从而得到该视频流数据对应的视频编码格式。因为同一数据格式的音频流数据或视频流数据，可能存在不同的编码格式，比如，音频流数据可以采用PCM（Pulse Code Modulation，脉冲编码调制）编码，也可以采用AC3（Audio Coding 3，音频编码3）编码，因此，需要确定音频流数据的音频编码格式，视频流数据的视频编码格式。Specifically, in the actual implementation process, audio and video data of the same data type, that is, a composite stream data format may have audio stream data in different formats and video stream data in different formats after decompositing, and the audio and video data can be determined first. After the data type of the stream data, obtain the encoding format corresponding to the data type of the audio stream data, so as to obtain the audio encoding format corresponding to the audio stream data; the same is true for the video stream part, first determine the data type of the video stream data , and then acquire the encoding format corresponding to the data type of the video stream data, so as to obtain the video encoding format corresponding to the video stream data. Because audio stream data or video stream data in the same data format may have different encoding formats, for example, audio stream data can use PCM (Pulse Code Modulation, pulse code modulation) encoding, can also use AC3 (Audio Coding 3. Audio encoding 3) Encoding. Therefore, it is necessary to determine the audio encoding format of the audio stream data and the video encoding format of the video stream data.

步骤S302.4：根据所述音频编码格式，确定对应的音频解码器；Step S302.4: Determine the corresponding audio decoder according to the audio encoding format;

步骤S302.5：根据所述视频编码格式，确定对应的视频解码器。Step S302.5: Determine the corresponding video decoder according to the video encoding format.

具体的，音频解码器和视频解码器均可以以插件的形式存储于预存插件注册表中，此处，根据音频编码格式确定音频流数据对应的音频解码器，即在所述预存插件注册表中，将音频编码格式与表中的数据格式匹配，找到解码功能的插件，即得到音频解码器，根据视频编码格式和所述预存插件注册表确定视频流数据对应的视频解码器，即在所述预存插件注册表中，将视频编码格式与表中的数据格式匹配，找到解码功能的插件，即得到视频解码器。比如，可以根据PCM编码格式或AC3编码格式确定对应的PCM解码器或AC3解码器，同样地，根据具体的视频编码格式确定对应的视频解码器。Specifically, both the audio decoder and the video decoder can be stored in the pre-stored plug-in registry in the form of plug-ins. Here, the audio decoder corresponding to the audio stream data is determined according to the audio encoding format, that is, in the pre-stored plug-in registry , match the audio encoding format with the data format in the table, find the plug-in of the decoding function, that is, obtain the audio decoder, determine the video decoder corresponding to the video stream data according to the video encoding format and the pre-stored plug-in registry, that is, in the In the pre-stored plug-in registry, match the video encoding format with the data format in the table, find the plug-in with decoding function, and obtain the video decoder. For example, the corresponding PCM decoder or AC3 decoder can be determined according to the PCM encoding format or the AC3 encoding format, and similarly, the corresponding video decoder can be determined according to the specific video encoding format.

步骤S303：通过所述音频解码器对所述音频流数据进行解码，得到音频；Step S303: Decoding the audio stream data by the audio decoder to obtain audio;

步骤S304：通过所述视频解码器对所述视频流数据进行解码，得到视频。Step S304: The video stream data is decoded by the video decoder to obtain a video.

具体的，投屏设备利用音频解码器对音频流数据进行解码，并利用视频解码器对视频流数据进行解码，即可得到待播放的音频和视频，其中，该音频和视频的长度一致，且与获取到的音视频数据的长度一样。Specifically, the screen projection device uses an audio decoder to decode the audio stream data, and uses a video decoder to decode the video stream data to obtain the audio and video to be played, wherein the audio and video have the same length, and It is the same length as the obtained audio and video data.

步骤S310：播放所述音频和所述视频。Step S310: Play the audio and the video.

具体的，投屏设备可以利用本地音频播放器和本地视频播放器或者音频播放插件和视频播放插件对音频和视频进行同步播放，保证音画同步。其中，音频播放插件和视频播放插件可以基于音频或视频的数据格式对应在预存插件注册表中查找和调取，以防止本地音频播放器和本地视频播放器出现不支持该音频或该视频的格式的情况出现。因此，在创建音视频处理线程时，可以设定多次利用预存插件注册表，调取不同过程需要使用的插件，以适应不同阶段，不同类型或不同数据格式的复合流数据、音频流数据、视频流数据、音频或视频等调取对应的插件，提高兼容性。Specifically, the screen projection device can use a local audio player and a local video player or an audio player plug-in and a video player plug-in to play audio and video synchronously to ensure synchronization of audio and video. Among them, the audio playback plug-in and video playback plug-in can be searched and retrieved in the pre-stored plug-in registry based on the data format of the audio or video, so as to prevent the local audio player and the local video player from appearing in a format that does not support the audio or the video situation occurs. Therefore, when creating an audio and video processing thread, you can set multiple times to use the pre-stored plug-in registry to call the plug-ins that need to be used in different processes, so as to adapt to different stages, different types or different data formats of composite stream data, audio stream data, Call the corresponding plug-in for video stream data, audio or video, etc., to improve compatibility.

在另一种实施方式中，步骤S310之前，所述方法还可以包括：In another implementation manner, before step S310, the method may further include:

步骤S320：构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器。Step S320: Build a multimedia architecture, the multimedia architecture includes an audio player and a video player.

进一步地，所述步骤S310可以包括：Further, the step S310 may include:

具体的，多媒体架构（GStreamer，用来构建流媒体应用的开源多媒体框架）可简化音/视频应用程序的开发。投屏设备构建该多媒体架构，可以减少音视频处理线程的线程段，利用该多媒体架构对音频或视频进行本地播放，具体为，根据目标解复合插件，对音视频数据进行解复合，得到音频流数据和视频流数据；对音频流数据和视频流数据分别进行类型分析，得到对应的音频编码格式和对应的视频编码格式；根据音频编码格式，确定对应的音频解码器，并根据视频编码格式，确定对应的视频解码器；根据音频解码器，对该音频流数据进行解码，得到音频，并根据视频解码器，对该视频流数据进行解码，得到视频；最后，利用控制该多媒体框架同时播放该音频和该视频。Specifically, the multimedia framework (GStreamer, an open source multimedia framework used to build streaming media applications) can simplify the development of audio/video applications. The multimedia architecture built by the screen projection device can reduce the thread segments of the audio and video processing threads, and use the multimedia architecture to play the audio or video locally. Specifically, according to the target decompositing plug-in, the audio and video data are decomplexed to obtain the audio stream data and video stream data; analyze the types of audio stream data and video stream data respectively, and obtain the corresponding audio encoding format and corresponding video encoding format; determine the corresponding audio decoder according to the audio encoding format, and according to the video encoding format, Determine the corresponding video decoder; decode the audio stream data according to the audio decoder to obtain audio, and decode the video stream data according to the video decoder to obtain video; finally, use the multimedia framework to simultaneously play the audio and this video.

需要说明的是，在基于步骤S240、S250、S280~S310的实施方式中更多实施细节可以参考实施例一中基于步骤S110~S160的实施方式中的描述，为了说明书的简洁，此处不再赘述。It should be noted that, for more implementation details in the implementation based on steps S240, S250, S280~S310, please refer to the description in the implementation based on steps S110~S160 in Example 1. repeat.

本实施例提供的音视频投屏方法，通过安全构建的实时流数据通道传送音视频数据，安全性高、实时性好；还通过针对音视频数据自行创建音视频处理线程，自定义性高，更贴合实际，方便针对终端设备的音视频格式采用最合理的插件进行处理，适应更多格式，提高投屏兼容性；通过在预存插件注册表中自动寻找合适的插件，自动创建音视频处理线程，不仅提高了投屏设备的兼容性，还在投屏延迟和卡顿方面可以有明显提高。The audio and video projection method provided in this embodiment transmits audio and video data through a securely constructed real-time streaming data channel, which has high security and good real-time performance; it also creates audio and video processing threads for audio and video data, which is highly customizable. It is more practical and convenient to use the most reasonable plug-in for processing the audio and video formats of the terminal equipment, adapt to more formats, and improve the compatibility of screen projection; automatically create audio and video processing by automatically searching for suitable plug-ins in the pre-stored plug-in registry Threads not only improve the compatibility of screen-casting devices, but also significantly improve screen-casting delay and stuttering.

实施例三Embodiment three

基于同一申请构思，参照图5，提出本申请音视频投屏装置的第一实施例，该音视频投屏装置可以为虚拟装置，应用于投屏设备。Based on the same application concept, referring to FIG. 5 , a first embodiment of the audio and video screen projection device of the present application is proposed. The audio and video screen projection device can be a virtual device and is applied to a screen projection device.

下面结合图5所示的功能模块示意图，对本实施例提供的音视频投屏装置进行详细描述，所述装置可以包括：The audio and video projection device provided in this embodiment will be described in detail below in conjunction with the functional module schematic diagram shown in FIG. 5 , and the device may include:

数据获取模块10，用于获取终端设备发送的音视频数据；Data obtaining module 10, is used for obtaining the audio-video data that terminal equipment sends;

类型分析模块20，用于基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；A type analysis module 20, configured to perform type analysis on the audio and video data based on a prestored type database to obtain a corresponding encoding format;

插件查找模块30，用于根据所述编码格式和预存插件注册表，获得目标解复合插件；The plug-in search module 30 is used to obtain the target decompound plug-in according to the encoding format and the pre-stored plug-in registry;

线程创建模块40，用于根据所述目标解复合插件，创建音视频处理线程；Thread creating module 40, used for decomposing plug-ins according to the target, creating audio and video processing threads;

数据处理模块50，用于基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；A data processing module 50, configured to perform demultiplexing processing on the audio and video data based on the audio and video processing thread to obtain audio and video;

投屏播放模块60，用于播放所述音频和所述视频。The screen-casting playing module 60 is configured to play the audio and the video.

进一步地，所述装置还可以包括：Further, the device may also include:

请求获取模块，用于获取所述终端设备发送的建立连接请求；A request obtaining module, configured to obtain a connection establishment request sent by the terminal device;

地址分配模块，用于响应所述建立连接请求，为所述终端设备分配网络协议地址；An address assignment module, configured to assign a network protocol address to the terminal device in response to the connection establishment request;

通道建立模块，用于根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道；A channel establishment module, configured to establish a real-time streaming channel between the terminal device and the screen projection device according to the network protocol address;

所述数据获取模块10具体用于：The data acquisition module 10 is specifically used for:

进一步地，所述类型分析模块20可以包括：Further, the type analysis module 20 may include:

数据格式获取单元，用于在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式；A data format acquiring unit, configured to search for a type corresponding to the audio-video data in a pre-stored type database to obtain the data format of the audio-video data;

编码格式获取单元，用于根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。The coding format obtaining unit is used for performing composite flow analysis on the audio and video data according to the data format of the audio and video data to obtain a corresponding coding format.

更进一步地，所述数据格式获取单元可以包括：Further, the data format acquisition unit may include:

字段获取子单元，用于获取所述音视频数据的报文字段；A field obtaining subunit, configured to obtain the message field of the audio and video data;

匹配子单元，用于将所述报文字段与预存的类型数据库中的数据类型进行匹配；A matching subunit, configured to match the message field with the data type in the pre-stored type database;

匹配结果子单元，用于若匹配成功，则得到所述报文字段对应的类型；The matching result subunit is used to obtain the type corresponding to the message field if the matching is successful;

数据格式确定子单元，用于将所述报文字段对应的类型确定为所述音视频数据的数据格式。The data format determining subunit is configured to determine the type corresponding to the message field as the data format of the audio and video data.

更进一步地，所述编码格式获取单元可以包括：Further, the encoding format obtaining unit may include:

字段类型获取子单元，用于根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系；The field type acquisition subunit is used to obtain the field type corresponding to the data format of the audio and video data according to the preset configuration rules, wherein the preset configuration rules include the corresponding relationship between each data format and each field type;

编码格式确定子单元，用于根据所述字段类型，确定所述音视频数据的编码格式。The encoding format determination subunit is configured to determine the encoding format of the audio and video data according to the field type.

插件预存模块，用于获取预存插件及其支持的数据格式；The plug-in pre-storage module is used to obtain the pre-storage plug-in and its supported data format;

注册表建立模块，用于建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。The registry building module is configured to establish a mapping relationship between the data format and the pre-stored plug-in, and obtain the pre-stored plug-in registry.

进一步地，所述插件查找模块30可以包括：Further, the plug-in search module 30 may include:

插件查找单元，用于在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。A plug-in search unit, configured to search a pre-stored plug-in registry for a decompositing plug-in corresponding to the encoding format to obtain a target decomposing plug-in, wherein the target decomposing plug-in is capable of performing composite stream data in the encoding format A plugin for decompositing.

进一步地，所述数据处理模块50可以包括：Further, the data processing module 50 may include:

第一线程段单元，用于根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据；The first thread segment unit is used to decompose the audio and video data according to the target decompounding plug-in to obtain audio stream data and video stream data;

第二线程段单元，用于对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器；The second thread segment unit is used to perform type analysis on the audio stream data and the video stream data respectively, and determine an audio decoder and a video decoder;

第三线程段单元，用于通过所述音频解码器对所述音频流数据进行解码，得到音频；A third thread segment unit, configured to decode the audio stream data through the audio decoder to obtain audio;

第四线程段单元，用于通过所述视频解码器对所述视频流数据进行解码，得到视频。The fourth thread segment unit is configured to use the video decoder to decode the video stream data to obtain a video.

更进一步地，所述第二线程段单元可以包括：Further, the second thread segment unit may include:

类型分析子单元，用于对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；A type analysis subunit, configured to perform type analysis on the audio stream data and the video stream data respectively, to obtain the data type of the audio stream data and the data type of the video stream data;

音频编码格式确定子单元，用于根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；An audio encoding format determination subunit, configured to determine an audio encoding format corresponding to the audio stream data according to the data type of the audio stream data;

视频编码格式确定子单元，用于根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式；A video encoding format determination subunit, configured to determine the video encoding format corresponding to the video stream data according to the data type of the video stream data;

音频解码器确定子单元，用于根据所述音频编码格式，确定对应的音频解码器；An audio decoder determining subunit, configured to determine a corresponding audio decoder according to the audio encoding format;

视频解码器确定子单元，用于根据所述视频编码格式，确定对应的视频解码器。The video decoder determining subunit is configured to determine a corresponding video decoder according to the video coding format.

多媒体模块，用于构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器；A multimedia module is used to build a multimedia framework, which includes an audio player and a video player;

所述投屏播放模块60具体用于：The screen projection playback module 60 is specifically used for:

需要说明，本实施例提供的音视频投屏装置中各个模块可实现的功能和对应达到的技术效果可以参照本申请音视频投屏方法各个实施例中具体实施方式的描述，为了说明书的简洁，此处不再赘述。It should be noted that the functions and corresponding technical effects achieved by each module in the audio and video screen projection device provided in this embodiment can refer to the description of specific implementations in each embodiment of the audio and video screen projection method in this application. For the sake of brevity of the description, I won't repeat them here.

实施例四Embodiment Four

基于同一申请构思，参照图2，为本申请各实施例涉及的投屏设备的硬件结构示意图。本实施例提供了一种投屏设备，所述投屏设备可以包括处理器和存储器，所述存储器中存储有音视频投屏程序，所述音视频投屏程序被所述处理器执行时，实现本申请音视频投屏方法各个实施例的全部或部分步骤。Based on the same application idea, refer to FIG. 2 , which is a schematic diagram of a hardware structure of a screen projection device involved in various embodiments of the present application. This embodiment provides a screen projection device. The screen projection device may include a processor and a memory. An audio and video projection program is stored in the memory. When the audio and video projection program is executed by the processor, Realize all or part of the steps of each embodiment of the audio and video projection method of the present application.

具体的，所述投屏设备是指能够实现网络连接的终端设备，可以是电视、会议平板、智能显示屏等终端设备。Specifically, the screen projection device refers to a terminal device capable of network connection, which may be a terminal device such as a TV, a conference tablet, or a smart display.

可以理解，所述设备还可以包括通信总线，用户接口和网络接口。It can be understood that the device may also include a communication bus, a user interface and a network interface.

其中，通信总线用于实现这些组件之间的连接通信。Among them, the communication bus is used to realize connection and communication between these components.

用户接口用于连接客户端，与客户端进行数据通信，用户接口可以包括输出单元，如显示屏、输入单元，如键盘，可选的，用户接口还可以包括其他输入/输出接口，比如标准的有线接口、无线接口。The user interface is used to connect the client and perform data communication with the client. The user interface may include an output unit, such as a display screen, and an input unit, such as a keyboard. Optionally, the user interface may also include other input/output interfaces, such as a standard Wired interface, wireless interface.

网络接口用于连接后台服务器，与后台服务器进行数据通信，网络接口可以包括输入/输出接口，比如标准的有线接口、无线接口，如Wi-Fi接口。The network interface is used to connect to the background server and perform data communication with the background server. The network interface may include an input/output interface, such as a standard wired interface, and a wireless interface, such as a Wi-Fi interface.

存储器用于存储各种类型的数据，这些数据例如可以包括该投屏设备中任何应用程序或方法的指令，以及应用程序相关的数据。存储器可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，例如静态随机存取存储器（Static Random Access Memory，简称SRAM），随机存取存储器（Random Access Memory，简称RAM），电可擦除可编程只读存储器（Electrically Erasable Programmable Read-Only Memory，简称EEPROM），可擦除可编程只读存储器（Erasable Programmable Read-Only Memory，简称EPROM），可编程只读存储器（Programmable Read-Only Memory，简称PROM），只读存储器（Read-Only Memory，简称ROM），磁存储器，快闪存储器，磁盘或光盘，可选的，存储器还可以是独立于所述处理器的存储装置。The memory is used to store various types of data, such data may include instructions of any application or method in the screen projection device, as well as application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or their combination, such as Static Random Access Memory (SRAM for short), Random Access Memory (Random Access Memory) Memory (RAM for short), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory) Erasable Programmable Read-Only Memory, referred to as EEPROM), Erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, referred to as EPROM), Programmable Read-Only Memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, ROM for short), magnetic memory, flash memory, magnetic disk or optical disk, and optionally, the memory may also be a storage device independent of the processor.

处理器用于调用存储器中存储的音视频投屏程序，并执行如上述的音视频投屏方法，处理器可以是专用集成电路（Application Specific Integrated Circuit，简称ASIC）、数字信号处理器（Digital Signal Processor，简称DSP）、数字信号处理设备（Digital Signal Processing Device，简称DSPD）、可编程逻辑器件（Programmable Logic Device，简称PLD）、现场可编程门阵列（Field Programmable Gate Array，简称FPGA）、控制器、微控制器、微处理器或其他电子元件，用于执行如上述音视频投屏方法各个实施例的全部或部分步骤。The processor is used to call the audio and video projection program stored in the memory, and execute the above-mentioned audio and video projection method. The processor can be an application specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), a digital signal processor (Digital Signal Processor (DSP for short), digital signal processing equipment (Digital Signal Processing Device, referred to as DSPD), programmable logic device (Programmable Logic Device, referred to as PLD), Field Programmable Gate Array (Field Programmable Gate Array (FPGA for short), controller, microcontroller, microprocessor or other electronic components are used to execute all or part of the steps in the various embodiments of the above-mentioned audio and video projection method.

实施例五Embodiment five

基于同一申请构思，本实施例提供了一种投屏***，所述***包括：Based on the same application idea, this embodiment provides a screen projection system, which includes:

终端设备和投屏设备；Terminal equipment and screen projection equipment;

其中，投屏设备的具体结构参照上述实施例，由于本实施例采用了上述所有实施例的全部技术方案，因此至少具有上述实施例的技术方案所带来的所有有益效果，在此不再一一赘述。Wherein, the specific structure of the screen projection device refers to the above-mentioned embodiments. Since this embodiment adopts all the technical solutions of all the above-mentioned embodiments, it at least has all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, and will not repeat them here. A repeat.

实施例六Embodiment six

基于同一申请构思，本实施例提供了一种计算机可读存储介质，如闪存、硬盘、多媒体卡、卡型存储器（例如，SD或DX存储器等）、随机访问存储器（RAM）、静态随机访问存储器（SRAM）、只读存储器（ROM）、电可擦除可编程只读存储器（EEPROM）、可编程只读存储器（PROM）、磁性存储器、磁盘、光盘、服务器、App应用商城等等，所述存储介质上存储有计算机程序，所述计算机程序可被一个或多个处理器执行，所述计算机程序被处理器执行时可以实现本申请音视频投屏方法各个实施例的全部或部分步骤。Based on the same application concept, this embodiment provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, server, App application store, etc., the A computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, all or part of the steps of the various embodiments of the audio and video projection method of the present application can be realized.

需要说明，上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。It should be noted that the serial numbers of the above embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

以上所述仅为本申请的可选实施例，并非因此限制本申请的专利范围，凡是在本申请的申请构思下，利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均包括在本申请的专利保护范围内。The above are only optional embodiments of the application, and do not limit the patent scope of the application. Under the application concept of the application, the equivalent structure or equivalent process transformation made by using the description of the application and the contents of the accompanying drawings, Or directly or indirectly used in other related technical fields, all are included in the patent protection scope of this application.

Claims

一种音视频投屏方法，其应用于投屏设备，所述方法包括：An audio and video screen projection method applied to a screen projection device, the method comprising:

获取终端设备发送的音视频数据；Obtain the audio and video data sent by the terminal device;

基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式；Performing type analysis on the audio and video data based on a pre-stored type database to obtain a corresponding encoding format;

根据所述编码格式和预存插件注册表，获得目标解复合插件；According to the encoding format and the pre-stored plug-in registration form, the target decompositing plug-in is obtained;

根据所述目标解复合插件，创建音视频处理线程；Create an audio and video processing thread according to the target decompounding plug-in;

基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频；以及Decompositing the audio and video data based on the audio and video processing thread to obtain audio and video; and

播放所述音频和所述视频。Play said audio and said video.
如权利要求1所述的音视频投屏方法，其中，所述获取终端设备发送的音视频数据的步骤之前，所述方法还包括：The audio and video projection method according to claim 1, wherein, before the step of acquiring the audio and video data sent by the terminal device, the method further comprises:

获取所述终端设备发送的建立连接请求；Obtain a connection establishment request sent by the terminal device;

响应所述建立连接请求，为所述终端设备分配网络协议地址；以及Allocating a network protocol address to the terminal device in response to the connection establishment request; and

根据所述网络协议地址，建立所述终端设备与所述投屏设备之间的实时流传输通道。Establish a real-time streaming channel between the terminal device and the screen projection device according to the network protocol address.
如权利要求2所述的音视频投屏方法，其中，所述获取终端设备发送的音视频数据包括：The audio and video projection method according to claim 2, wherein said acquiring the audio and video data sent by the terminal device comprises:

获取所述终端设备通过所述实时流传输通道发送的音视频数据。Acquiring audio and video data sent by the terminal device through the real-time streaming channel.
如权利要求1所述的音视频投屏方法，其中，所述基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式包括：The audio and video screen projection method according to claim 1, wherein the type analysis of the audio and video data based on the pre-stored type database, and obtaining the corresponding encoding format includes:

在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式；以及Find the type corresponding to the audio-video data in a pre-stored type database to obtain the data format of the audio-video data; and

根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式。According to the data format of the audio and video data, the composite flow analysis is performed on the audio and video data to obtain a corresponding encoding format.
如权利要求4所述的音视频投屏方法，其中，所述在预存的类型数据库中查找与所述音视频数据对应的类型，得到所述音视频数据的数据格式包括：The audio and video projection method according to claim 4, wherein said searching the type corresponding to the audio and video data in the pre-stored type database, and obtaining the data format of the audio and video data includes:

获取所述音视频数据的报文字段；Obtain the message field of the audio and video data;

将所述报文字段与预存的类型数据库中的数据类型进行匹配；Matching the message field with the data type in the pre-stored type database;

若所述报文字段与预存的类型数据库中的数据类型匹配成功，则得到所述报文字段对应的类型；以及If the message field matches the data type in the pre-stored type database successfully, then obtain the type corresponding to the message field; and

将所述报文字段对应的类型确定为所述音视频数据的数据格式。The type corresponding to the message field is determined as the data format of the audio and video data.
如权利要求4所述的音视频投屏方法，其中，所述根据所述音视频数据的数据格式，对所述音视频数据进行复合流分析，得到对应的编码格式包括：The audio and video projection method according to claim 4, wherein, according to the data format of the audio and video data, performing composite flow analysis on the audio and video data to obtain a corresponding encoding format includes:

根据预设配置规则，获得与所述音视频数据的数据格式对应的字段类型，其中，所述预设配置规则包含各数据格式与各字段类型的对应关系；以及Obtain the field type corresponding to the data format of the audio and video data according to a preset configuration rule, wherein the preset configuration rule includes a correspondence between each data format and each field type; and

根据所述字段类型，确定所述音视频数据的编码格式。Determine the encoding format of the audio and video data according to the field type.
如权利要求1所述的音视频投屏方法，其中，所述根据所述编码格式和预存插件注册表，获得目标解复合插件的步骤之前，所述方法还包括：The audio and video projection method according to claim 1, wherein, before the step of obtaining the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry, the method further includes:

获取预存插件及其支持的数据格式；以及Get pre-existing plugins and their supported data formats; and

建立所述数据格式与所述预存插件之间的映射关系，获得所述预存插件注册表。A mapping relationship between the data format and the pre-stored plug-in is established to obtain the pre-stored plug-in registry.
如权利要求1所述的音视频投屏方法，其中，所述根据所述编码格式和预存插件注册表，获得目标解复合插件包括：The audio and video projection method according to claim 1, wherein said obtaining the target decompositing plug-in according to the encoding format and the pre-stored plug-in registry includes:

在预存插件注册表中查找与所述编码格式对应的解复合插件，获得目标解复合插件，其中，所述目标解复合插件为可对所述编码格式的复合流数据进行解复合处理的插件。Search the pre-stored plug-in registry for a decomplexing plug-in corresponding to the encoding format to obtain a target decomposing plug-in, wherein the target decomposing plug-in is a plug-in capable of performing decomplexing processing on composite stream data in the encoding format.
如权利要求1所述的音视频投屏方法，其中，所述基于所述音视频处理线程对所述音视频数据进行解复合处理，获得音频和视频包括：The audio and video screen projection method according to claim 1, wherein said performing decomplexing processing on said audio and video data based on said audio and video processing thread, and obtaining audio and video comprises:

根据所述目标解复合插件，对所述音视频数据进行解复合，得到音频流数据和视频流数据；Decompounding the audio and video data according to the target decompounding plug-in to obtain audio stream data and video stream data;

对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器；Perform type analysis on the audio stream data and the video stream data respectively, and determine an audio decoder and a video decoder;

通过所述音频解码器对所述音频流数据进行解码，得到音频；以及Decoding the audio stream data by the audio decoder to obtain audio; and

通过所述视频解码器对所述视频流数据进行解码，得到视频。The video stream data is decoded by the video decoder to obtain a video.
如权利要求9所述的音视频投屏方法，其中，所述对所述音频流数据和所述视频流数据分别进行类型分析，确定音频解码器和视频解码器包括：The audio and video projection method according to claim 9, wherein said performing type analysis on said audio stream data and said video stream data respectively, and determining an audio decoder and a video decoder comprises:

对所述音频流数据和所述视频流数据分别进行类型分析，得到所述音频流数据的数据类型和所述视频流数据的数据类型；Perform type analysis on the audio stream data and the video stream data respectively to obtain the data type of the audio stream data and the data type of the video stream data;

根据所述音频流数据的数据类型，确定所述音频流数据对应的音频编码格式；According to the data type of the audio stream data, determine the audio encoding format corresponding to the audio stream data;

根据所述视频流数据的数据类型，确定所述视频流数据对应的视频编码格式；According to the data type of the video stream data, determine the video encoding format corresponding to the video stream data;

根据所述音频编码格式，确定对应的音频解码器；以及Determine a corresponding audio decoder according to the audio encoding format; and

根据所述视频编码格式，确定对应的视频解码器。According to the video coding format, a corresponding video decoder is determined.
如权利要求1所述的音视频投屏方法，其中，所述播放所述音频和所述视频的步骤之前，所述方法还包括：The audio and video projection method according to claim 1, wherein, before the step of playing the audio and the video, the method further comprises:

构建多媒体架构，所述多媒体架构包括音频播放器和视频播放器。A multimedia architecture is constructed, and the multimedia architecture includes an audio player and a video player.
如权利要求11所述的音视频投屏方法，其中，所述播放所述音频和所述视频包括：The audio and video projection method according to claim 11, wherein said playing said audio and said video comprises:

通过所述多媒体架构同步播放所述音频和所述视频。The audio and the video are played synchronously through the multimedia framework.
如权利要求1所述的音视频投屏方法，其中，所述基于预存的类型数据库对所述音视频数据进行类型分析，获得对应的编码格式包括：The audio and video screen projection method according to claim 1, wherein the type analysis of the audio and video data based on the pre-stored type database, and obtaining the corresponding encoding format includes:

在预存的类型数据库中查找到所述音视频数据的数据格式；以及Find the data format of the audio and video data in the pre-stored type database; and

根据所述数据格式进行编码格式的匹配，获得所述音视频数据对应的编码格式。The encoding format is matched according to the data format to obtain the encoding format corresponding to the audio and video data.
一种投屏设备，其包括：A screen projection device, comprising:

存储器，存储有音视频投屏程序；以及A memory storing an audio and video projection program; and

处理器，执行所述音视频投屏程序，以实现如权利要求1至13中任一项所述的音视频投屏方法。The processor executes the audio and video screen projection program, so as to realize the audio and video screen projection method according to any one of claims 1 to 13.
一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序可被一个或多个处理器执行，以实现如权利要求1至13中任一项所述的音视频投屏方法。A computer-readable storage medium, on which a computer program is stored, and the computer program can be executed by one or more processors, so as to realize the audio and video projection method according to any one of claims 1 to 13.