CN109769124A - Mixed flow method, apparatus, electronic equipment and storage medium - Google Patents
Mixed flow method, apparatus, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109769124A CN109769124A CN201811526963.XA CN201811526963A CN109769124A CN 109769124 A CN109769124 A CN 109769124A CN 201811526963 A CN201811526963 A CN 201811526963A CN 109769124 A CN109769124 A CN 109769124A
- Authority
- CN
- China
- Prior art keywords
- time
- data packet
- media stream
- mapping
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
This application discloses a kind of mixed flow method, apparatus, electronic equipment and storage mediums, are related to audio-video processing technology field.Wherein, this method comprises: receiving multipath media stream；The acquisition time of the multipath media stream is mapped to same object time axis respectively, obtains the mapping time of corresponding different media flows；It is mapped to the mapping time that the object time axis obtains according to the multipath media stream, mixed flow is carried out to the multipath media stream.The road Hou Ge Media Stream is mapped in this programme has unified time standard, makes the effectively synchronous alignment of each road Media Stream of mixed flow.
Description
Technical field
This application involves audio-video processing technology fields, more particularly, to a kind of mixed flow method, apparatus, electronic equipment
And storage medium.
Background technique
With the development of science and technology, more people's net casts become increasingly prevalent.But current more people's net casts
During, usually there are problems that the audio-video at each line end can not effectively synchronize alignment.
Summary of the invention
In view of the above problems, present applicant proposes a kind of mixed flow method, apparatus, electronic equipment and storage mediums, to improve
The above problem.
In a first aspect, the embodiment of the present application provides a kind of mixed flow method, applied to the mixed flow end of Media Stream, the method
It include: reception multipath media stream；The acquisition time of the multipath media stream is mapped to same object time axis, acquisition pair respectively
Answer the mapping time of different media flows；It is mapped to the mapping time that the object time axis obtains according to the multipath media stream,
Mixed flow is carried out to the multipath media stream.
Second aspect, the embodiment of the present application provide a kind of flow mixing device, and described device includes: receiving module, for connecing
Receive multipath media stream；Mapping block, for the acquisition time of the multipath media stream to be mapped to same object time axis respectively,
Obtain the mapping time of corresponding different media flows；Mixed flow module, when for being mapped to the target according to the multipath media stream
Between the obtained mapping time of axis, mixed flow is carried out to the multipath media stream.
The third aspect, the embodiment of the present application provide a kind of electronic equipment, comprising: one or more processors；Memory；
One or more programs.Wherein one or more of programs are stored in the memory and are configured as by one
Or multiple processors execute, one or more of programs are configured to carry out above-mentioned method.
Fourth aspect, the embodiment of the present application provide a kind of computer readable storage medium, the computer-readable storage
Program code is stored in medium, said program code can be called by processor and execute above-mentioned method.
Mixed flow method, apparatus, electronic equipment and storage medium provided by the embodiments of the present application, by the acquisition of multipath media stream
Time map is mapped to the mapping that object time axis obtains to same object time axis, further according to the acquisition time of each road Media Stream
Time carries out the mixed flow of each road Media Stream.It is marked after the acquisition time mapping of the road this programme Zhong Ge Media Stream with the unified time
Standard makes the effectively synchronous alignment of each road Media Stream of mixed flow.
Detailed description of the invention
In order to more clearly explain the technical solutions in the embodiments of the present application, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, the drawings in the following description are only some examples of the present application, for
For those skilled in the art, without creative efforts, it can also be obtained according to these attached drawings other attached
Figure.
Fig. 1 shows the schematic diagram of audio/video player system provided by the embodiments of the present application.
Fig. 2 shows the flow charts for the mixed flow method that one embodiment of the application provides.
Fig. 3 shows the flow chart for the mixed flow method that another embodiment of the application provides.
Fig. 4 shows the flow chart of a part of step in the mixed flow method that another embodiment of the application provides.
The data packet that Fig. 5 shows Media Stream in the embodiment of the present application sends schematic diagram；
Fig. 6 shows the flow chart of another part step in the mixed flow method that another embodiment of the application provides.
Fig. 7 shows the flow chart of another part steps in the mixed flow method that another embodiment of the application provides.
Fig. 8 shows a kind of display schematic diagram provided by the embodiments of the present application；
Fig. 9 shows another display schematic diagram provided by the embodiments of the present application；
Figure 10 shows the functional block diagram of flow mixing device provided by the embodiments of the present application.
Figure 11 shows the structural block diagram of electronic equipment provided by the embodiments of the present application.
Figure 12 is the embodiment of the present application for saving or carrying the mixed flow method realized according to the embodiment of the present application
The storage unit of program code.
Specific embodiment
In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application
Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described.
Mixed flow method, apparatus, electronic equipment and storage medium provided by the embodiments of the present application can be applied to video playing
System, the audio/video player system include that server and multiple terminals, different terminals are connect with server network.Wherein, it services
Device can be server cluster, terminal can be mobile phone, computer, plate or other can obtain real-time video simultaneously from server
The smart machine of broadcasting.In the audio/video player system, the quantity of terminal is not intended to limit, and can be two or more.
In the embodiment of the present application, which can be live broadcast system, as shown in Figure 1, include main broadcaster end with
And viewer end, it is used for network direct broadcasting.Wherein, main broadcaster end, that is, corresponding terminal of main broadcaster, the corresponding terminal of viewer end, that is, spectators.When this
Audio/video player system be live broadcast system when, can also include content distributing network (Content Delivery Network,
CDN), for pushing Media Stream to viewer end.
In the embodiment of the present application, which can be used for more people's net casts under live scene, such as connect
Wheat interaction live streaming.Be used for more people's net casts when, mixed flow can carry out at any one main broadcaster end, can server into
Row, can also carry out in viewer end.
As an implementation, mixed flow can carry out at a main broadcaster end, and from server pull, other are led at the main broadcaster end
The Media Stream that end is sent to server is broadcast, the Media Stream pulled and local collected media data mixed flow obtain mixed flow
The single stream obtained is sent to CDN network, so that viewer end pulls broadcasting from CDN network.Wherein, single stream that mixed flow obtains, is multichannel
Media Stream all the way is obtained after Media Stream mixed flow.For example, the using in Fig. 1 first main broadcaster end is as the main broadcaster end of mixed flow, the first main broadcaster end
The two-way Media Stream pulled can be adopted with local from the Media Stream at server pull the second main broadcaster end and third main broadcaster end
Single stream after mixed flow is sent CDN network by the media data mixed flow of collection.Viewer end can pull this from CDN network and singly flow into
Row plays.In addition, in this embodiment, the main broadcaster end for carrying out mixed flow can also be using the media data locally acquired as all the way
Media Stream is pushed to server.
As an implementation, mixed flow can be carried out in server.Each main broadcaster end takes to server push Media Stream
Device be engaged in for after the multipath media stream mixed flow received, single stream that mixed flow obtains is pushed to CDN network, so that viewer end is from CDN
Network pulls broadcasting.For example, in Fig. 1 the first main broadcaster end, the second main broadcaster end and third main broadcaster end to server push Media Stream,
Server is sent to CDN network after carrying out mixed flow to the three road Media Streams received.Viewer end can pull mixed flow from CDN network
Single stream afterwards plays out.
As an implementation, mixed flow can also be carried out in viewer end.Each main broadcaster end to server push Media Stream,
The multipath media stream received is sent CDN network by server.Each viewer end can pull the multi-path media from CDN network
Stream plays after the multipath media stream is carried out mixed flow.
In the embodiment of the present application, audio/video player system is not limited to above-mentioned network direct broadcasting scene, can also apply
The usage scenario of Media Stream mixed flow is carried out in other any need, the embodiment of the present application will not enumerate.
Defining terminal or the server for carrying out mixed flow to Media Stream is mixed flow end.It should be understood that mixed flow end is carrying out
Before mixed flow, the Media Stream received is decoded.If single stream after mixed flow is sent, encoded before sending.
Specific decoding process and cataloged procedure the embodiment of the present application and without repeating.
Wherein, when to Media Stream mixed flow, how synchronously to mix multiple Media Streams is the key that mixed flow skill
Art.It is coherent synchronization that mixed flow, which requires the interaction between a plurality of Media Stream, and audio-video is alignment.However due between Media Stream
Time shaft be usually inconsistent, in addition generate the Media Stream encoding time delay, sending strategy and the transmission jitter of terminal etc.
The influence of factor synchronizes alignment to multipath media stream and brings very big challenge.Wherein, the movement of object be all in the time and
It is carried out in space, usually with the moment is a little indicated on time shaft, indicates the time with time shaft the preceding paragraph distance.The different time
Axis indicates different time standards, and the substandard time shaft of different time is different.The a certain moment under same time standard, such as
At a certain moment in universal time, the moment is different on axis in different times.
For example, when carrying out mixed flow, if using the acquisition time of each Media Stream as the foundation being aligned between Media Stream, by
May be inconsistent in the terminal time axis for generating and sending Media Stream, when such as more people's net casts, the time shaft at each main broadcaster end
There are difference, the mutual time shaft of the acquisition time carried in each road Media Stream may be different, thus in mixed flow foundation when
Between be inaccurate.For example, sending the time shaft ratio UTC (Greenwich mean time of the terminal of first via Media Stream for two-way Media Stream
Between) 1 minute fast, time shaft ratio UTC slow 1 minute of the second road Media Stream is sent, for being in UTC simultaneously in this two-way Media Stream
The data packet of the data of 2 points of acquisitions in 15 minutes, then the acquisition time that the data packet carries in first via Media Stream be 2 points 16 minutes, the
In two road Media Streams the data packet carry acquisition time be 2 points 14 minutes.Then in mixed flow, by first via Media Stream 2: 14
2 points of 14 minutes data in the second Media Stream of data mixing divided, it is 2 points of 15 minutes numbers that reality, which is by UTC in first via Media Stream,
It is 2 points of 17 minutes data according to UTC in the second road Media Stream of mixing.
If receiving time of Media Stream as the foundation being aligned between Media Stream using mixed flow termination, due to encoding time delay,
The factors such as sending strategy and transmission jitter, for different terminals in collected Media Stream of same time, the time for reaching mixed flow end can
It can be different.
The accurate synchronization of each Media Stream when in order to make mixed flow, can by each terminal for generating and sending Media Stream when
Between axis it is synchronous be NTP (Network Time Protocol) time shaft, so that the clock of terminal is synchronized to UTC (world standard generation
Boundary).Inventors have found that the time shaft method of synchronization depends on each terminal NTP time synchronization accuracy, and need each terminal corresponding
It supports, logic is complicated, poor universality.
In view of the above-mentioned problems, inventor after study, proposes mixed flow method, apparatus provided by the embodiments of the present application, electricity
Sub- equipment and storage medium, for the acquisition time for needing the multipath media stream of mixed flow to be mapped to same time shaft, further according to
It maps obtained mapping time and mixed flow, the accuracy being aligned when improving mixed flow is carried out to each road Media Stream.
It below will be by specific embodiment to mixed flow method, apparatus, electronic equipment and storage provided by the embodiments of the present application
Medium is described in detail.
Referring to Fig. 2, showing mixed flow method provided by the embodiments of the present application.Specifically, this method is for Media Stream
One end that mixed flow end, i.e. reception Media Stream carry out mixed flow, can be server or terminal.Specifically, this method comprises:
Step S110: multipath media stream is received.
If mixed flow is realized by terminal, and needs to mix local Media Stream when mixed flow, then the received multipath media stream of terminal
The Media Stream of the Media Stream and local that are sent including other terminals.Wherein, the local Media Stream i.e. audio-video of the terminal is adopted
Collect the collected audio data of equipment and video data.
If mixed flow is realized by terminal, and only mixes the Media Stream of other terminals received when mixed flow, then terminal is received
Multipath media stream is the Media Stream of other terminals.
If mixed flow is realized by server, received multipath media stream is sent by each terminal.
Step S120: the acquisition time of the multipath media stream is mapped to same object time axis respectively, is corresponded to
The mapping time of different media flows.
Each road Media Stream received, acquisition time is respectively to acquire time of the terminal of the media data of the Media Stream
Axis is as time shaft.The accurate alignment of each Media Stream when for mixed flow, by the acquisition time of the multipath media stream received into
Row time shaft is unified, and specifically the acquisition time of each road Media Stream can be mapped to same time shaft, defines this be mappeding to
Time shaft is object time axis, defines acquisition time and is mapped to the time of object time axis as mapping time.
Step S130: the mapping time that the object time axis obtains is mapped to according to the multipath media stream, to described
Multipath media stream carries out mixed flow.
After the acquisition time of each road Media Stream is mapped to object time axis, the mapping time of each road Media Stream, then root are obtained
Mixed flow is carried out according to the mapping time of each road Media Stream.The mapping time of each road Media Stream is unified using object time axis as standard, mixes
It is aligned when stream more acurrate.
In the embodiment of the present application, the Media Stream received is carried out time shaft mapping by the mixed flow end of Media Stream, by each road matchmaker
The acquisition time of body stream is uniformly mapped to object time axis, obtains the corresponding mapping time of each road Media Stream, to make each
Standard based on the time of road Media Stream is consistent, and it is more accurate to carry out alignment synchronization when mixed flow.
Another embodiment of the application provides a kind of mixed flow method, by mapping the acquisition time of data packet in Media Stream
To object time axis, realize that the acquisition time of Media Stream is mapped to object time axis.Specifically, this method can be applied to service
Device or terminal.For example, if mixed flow is realized by main broadcaster end, party's rule can be applied under the scene of more people's net casts
Main broadcaster end；If mixed flow is realized by server, this method can be applied to server.
As audio-video connects the development of wheat application under live streaming media scene, server-side audio mixing mixes picture, i.e. mixed flow, due to
Advantage in terms of delay, bandwidth, scalability is more and more important in the effect being broadcast live in weak interactive service, the quality of server mixed flow
Determine the user experience of live streaming.The embodiment of the present application is mainly under more people's net cast scenes, server is as mixed flow end
Example is illustrated, it is possible to understand that, the embodiment of the present application is not limited to more people's net cast scenes, is also not necessarily limited to server.Please
Referring to Fig. 3, this method comprises:
Step S210: multipath media stream is received, wherein every road Media Stream includes multiple data packets.
The media data of acquisition according to time order and function partition packetization, will be packaged the data packet obtained and is sent to service by terminal
Device forms stable and continuous stream, the i.e. Media Stream of terminal transmission.The media data includes audio data and video counts
According to.
Wherein, when media data is packaged as data packet, the acquisition time that the media data is added in the packet is made
For the acquisition time (PTS) of media data in data packet, which is the audio collecting device and video acquisition of terminal
Equipment collects the time of respective media data, and be acquire the media data terminal time shaft on time.For example,
2 points on terminal time axis of audio data and video data in data packet collect for 15 minutes, then are allocated as 2: 15 to adopt
The collection time is added in data packet.Specifically, the field for indicating acquisition time can be set in data packet, the field by
Acquisition time filling.
In addition, optional, terminal can also add in each data packet when sending data packet and send the data packet
Time.It does not limit, such as can be, data packet in the embodiment of the present application in the concrete mode of data packet addition sending time
The middle field being arranged for indicating sending time fills the field with current time, when sending data packet to make data
The sending time of data packet is carried in packet.It should be understood that terminal is when adding current time for data packet, the time of addition is
Using the time shaft of terminal as the time of standard.For example, terminal, when sending data packet, current time is 2 points on terminal time axis
15 points, then 2: 15 are allocated as being filled into data packet for sending time.
Server receives the multipath media stream sent by different terminals, and every road Media Stream is sent by a terminal.
Step S220: the acquisition time of media data in each data packet is mapped to the object time axis respectively, is obtained
Obtain the corresponding mapping time of each data packet.The acquisition time is the time on corresponding terminal time axis, the mapping
Time is that the acquisition time of media data in each data packet is mapped to the time of the object time axis.
The acquisition time of multipath media stream is mapped to same object time axis by server, specifically can be each road media
The acquisition time of media data is mapped to object time axis in data packet in stream.Wherein, media data is adopted in each data packet
Collect time map to the time of object time axis, is defined as the mapping time of the data packet.
Specifically, as shown in figure 4, the step may include:
Step S221: the acquisition time of media data in each data packet is obtained.
The acquisition time of media data in the available data packet received of server.Wherein, server can connect
When receiving a data packet, then the acquisition time of media data in the data packet is obtained.Media in the data packet that server obtains
The acquisition time of data is the time in the time shaft for the terminal for sending the data packet.
Server can obtain the acquisition time of wherein media data from data packet.Wherein, the concrete mode of acquisition can
It is determined in a manner of adding acquisition time according to data packet.For example, terminal in the packet fill in the data packet by certain field
The acquisition time of media data, the then acquisition time of server field read data packet from the data packet received.
Step S222: for each data packet, reflecting between the object time axis and corresponding terminal time axis is obtained
Penetrate value.
For any one data packet, the acquisition time of media data in data packet is if desired mapped to the object time
Axis then can first obtain the mapping value between object time axis terminal time axis corresponding with the data packet.Wherein, data packet pair
The terminal time axis answered, that is, send the time shaft of the terminal of the data packet.Mapping between object time axis and terminal time axis
Value needs the value of corresponding conversion when as by the time map on terminal time axis to object time axis.Specifically, this is reflected
Penetrate time difference of the value between object time axis and terminal time axis.
For different media flows, it is sent to server from different terminals, reaches time of server by server according to clothes
Being engaged in, device time shaft is determining, and therefore, each road Media Stream can all be generated with server time axis and centainly be contacted.The server time axis is
The time shaft of server.If by the time map of each Media Stream to server time axis, the time map to server time
Axis is the time plus the time difference between terminal time axis and server time axis.For example, server time axis is indicated with TS,
TA indicates the time shaft of certain terminal A, then between server time axis and the time shaft of terminal A have time difference TS-TA, this when
Between difference indicate under unified time standard, such as on any one moment of universal time, server time axis when
Between difference with the time on terminal A time shaft.Any one data packet in Media Stream issued for terminal A, if wherein
Including some time on terminal time axis be TA1, then being mapped to server time axis is then TA1+ (TS-TA).Wherein,
TS-TA is unknown.In the embodiment of the present application, mapping value can be calculated with time of some in data packet on terminal time axis,
The time defined on the terminal time axis is the object time, which can be acquisition time, be also possible to data packet
Sending time.
In actual scene, Media Stream is sent to server with time delay from terminal, that is to say, that Media Stream is sent out from terminal
Certain time is needed to the process for reaching server out, which is that the spent time is transmitted between terminal to server.
For each data packet, time delay is the time issued from terminal to reaching between server.Then for every in Media Stream
A data packet, issues to by server from terminal and receives, in the receiving time on server time axis and on terminal time axis
Sending time between time difference, equal to the time difference between server time axis and terminal time axis plus the data packet
Time delay.For example, from terminal A issue Media Stream in some data packet, if its sending time be tA1, reach server when
Between be TS1, time delay DelayAS1, then TS1-tA1=TS-TA+DelayAS1.
The corresponding time delay of each data packet on Media Stream, when data packet is more and more on Media Stream all the way, i.e., from
The data packet that some terminal is sent to server is more and more, and the corresponding minimal time delay of the Media Stream will gradually restrain, and to this
The ideal time delay of Media Stream restrains, also, is proved by test, and convergence rate meets display demand.The corresponding minimum of Media Stream
All data packets currently transmitted on time delay, that is, Media Stream when Yanzhong minimal time delay, the ideal time delay of Media Stream indicates
The minimal time delay that server and terminal can achieve.
Corresponding, the minimum time on Media Stream between the receiving time and sending time of all data packets is poor, will be by
The time difference between server time axis and terminal time axis is walked plus ideal time delay convergence.
Such as shown in Fig. 5, from terminal A issue Media Stream A sent n data packet, be followed successively by data packet 1,2,3 to
Data packet n, sending time of each data packet on the time shaft of terminal A is respectively tA1, tA2, tA3 to tAn, in server
Receiving time on time shaft is respectively TS1, TS2, TS3 to TSn, corresponding time delay be respectively DelayAS1, DelayAS2,
DelayAS3 to DelayASn.Wherein, time difference 1 '=TS1-tA1=of Δ between 1 receiving time of data packet and sending time
TS-TA+DelayAS1；Time difference 2 '=TS2-tA2=TS-TA+ of Δ between 2 receiving time of data packet and sending time
DelayAS2；Time difference 3 '=TS3-tA3=TS-TA+DelayAS3 of Δ between 3 receiving time of data packet and sending time,
Until time difference Δ n '=TSn-tAn=TS-TA+DelayASn between data packet n receiving time and sending time.
Indicate the corresponding minimal time delay of Media Stream A with Delaymin, i.e. Delaymin=min (DelayAS1,
DelayAS2, DelayAS3 ... DelayASn).The ideal time delay of Media Stream A is indicated with DelayAS.Wherein, with n numerical value
Increase, i.e., the packets increase sent from terminal A to server, the packets increase that Media Stream A includes, Delaymin will gradually
It is restrained to DelayAS, the numerical value of n is bigger, more converges on DelayAS.Connecing for all data packets on Media Stream is indicated with Δ min '
Minimum time between time receiving between sending time is poor, i.e. Δ min '=min (Δ 1 ', Δ 2 ', Δ 3 ' ... Δ n '), Δ min '
=TS-TA+Delaymin.Then on Media Stream A data packet increase, Δ min ' will gradually be restrained to TS-TA+DelayAS.
It is not much different for ideal time delay, of substantially equal different media flows, server time axis adds corresponding ideal
Time delay is of substantially equal.For example, for Media Stream A and Media Stream B, corresponding ideal time delay is respectively DelayAS, DelayBS,
DelayAS is substantially equal to DelayBS, then TS+DelayAS is substantially equal to TS+DelayBS, also, the experiment proved that, wherein can
Small difference existing for energy can be ignored.
Therefore, the different media flows of substantially equal for ideal time delay, if adding corresponding ideal with server time axis
Time delay is as object time axis, then the minimum time difference on Media Stream between the receiving time and sending time of data packet converges on
Time difference between object time axis and terminal time axis.That is, for Media Stream A, if being added with server time axis
Corresponding ideal time delay TS+DelayAS is as object time axis, then the time difference between object time axis and terminal time axis is
TS+DelayAS-TA.In Media Stream, the corresponding Δ min ' of each data packet is computable, and Δ min ' is to TS-TA
+ DelayAS convergence, therefore, can be using corresponding Δ min ' as object time axis and terminal time for each data packet
Mapping value between axis, also, with the packets increase on Media Stream, i.e., server is already sent to before some data packet
Data packet number increase, Δ min ' will become closer to the time difference TS+ between object time axis and terminal time axis
DelayAS-TA。
Therefore, the different media flows of substantially equal for ideal time delay can add corresponding reason with server time axis
Think time delay as object time axis.That is, for Media Stream A, can using TS+DelayAS as object time axis, for
Media Stream B, can be using TS+DelayBS as object time axis.For any one data packet m on Media Stream A, adopted
It, can be using the corresponding Δ min ' of data packet m as mapping value when collection time t ' A (m) is mapped to object time axis.
Also, the terminal and server met certain condition, when being sent to the ideal of the Media Stream of server from terminal
Prolong of substantially equal.For example, the condition is the terminal and server in the same area.It was proved that same national region
Interior terminal and server, the Media Stream ideal time delay for being sent to server from terminal are all of substantially equal.As field is broadcast live in more people
Under scape, the main broadcaster end in CHINESE REGION sends audio, video data to server, the ideal time delay base of the Media Stream at each main broadcaster end
This is equal.
Therefore, in the embodiment of the present application, for any one data packet, when obtaining object time axis with corresponding terminal
Between mapping value between axis when, it is poor to calculate minimum time according to the sending time and receiving time of data packet, with the number
According to the corresponding minimum time difference of packet as the mapping value.
In addition, in actual scene, data in data packet have between the receiving time from acquisition time to server
Time difference, the time difference are equal to the time difference between the sending time in receiving time and data packet plus acquisition time to transmission
Time difference between time.Acquisition time in data packet is sent to the time difference between sending time, for data from collecting
Between the processing time, by data from collect send between treatment process influenced, such as coding, packing, different data
Packet, the corresponding processing time are different.As data packet is more and more in Media Stream, i.e., server is sent to from some terminal
Data packet is more and more, and the Media Stream corresponding minimum treat time will gradually restrain, and when to the ideal process of the Media Stream
Between restrain, also, prove that convergence rate meets display demand by test.The corresponding minimum treat time, that is, media of Media Stream
Minimum treat time in the processing time for all data packets currently transmitted on stream, the ideal process timetable of Media Stream
Show the corresponding terminal of the Media Stream by audio, video data from collect send between the minimum treat time that can achieve.
Then corresponding, the acquisition time on Media Stream in all data packets is poor to the minimum time between receiving time,
Ideal time delay gradually will be added to the time difference between server time axis and terminal time axis, along with the ideal process time receives
It holds back.
In addition, different terminals, the corresponding ideal process time is of substantially equal, and difference therein is to data in mixed flow
Alignment influences little.Therefore, in the embodiment of the present application, for different Media Streams, can with server time axis plus pair
The ideal time delay answered is along with the corresponding ideal process time is as object time axis.Acquisition on Media Stream in all data packets
Time is poor to the minimum time between receiving time, to the time difference convergence between object time axis and terminal time axis, then may be used
With the acquisition time in data packets all using on Media Stream to the minimum time difference between receiving time as mapping value.
Therefore, in the embodiment of the present application, mapping value can be calculated with acquisition time or sending time.Definition is based on
The time for calculating mapping value is the object time, i.e. the object time can be acquisition time, be also possible to sending time.It specifically can be with
The corresponding receiving time of data packet and minimum time difference between the object time are as mapping value.Wherein, data packet is corresponding
Minimum time between receiving time and object time is poor, when being the reception of multiple data packets where data packet in Media Stream
Between and the object time between the time difference minimum value.
Specifically, as an implementation, in order to improve processing speed, for currently processed data packet, Ke Yicong
It is poor that minimum time is obtained in the data packet corresponding time difference before the data packet, is reflected as currently processed data packet is corresponding
Penetrate value.Specifically, referring to Fig. 6, in this embodiment, step S222 may include:
Step S2221: for each data packet, obtaining in the Media Stream including the data packet, more before the data packet
A data packet corresponding time difference, obtain multiple time differences.Wherein, each time difference is corresponding data packet at mixed flow end
Receiving time on time shaft and the difference between the object time on terminal time axis.
Wherein, if using sending time as the object time, the mode that server obtains the sending time of data packet can root
It is determined according to the mode of data packet addition sending time.For example, terminal in the packet fill when data packet is sent by certain field
Sending time, then server is from the sending time for receiving the field read data packet in data packet.
Step S2224: the minimum value in the multiple time difference is obtained.
Step S2227: using the minimum value as the mapping value.
For currently processed data packet, each data packet before, which can correspond to, obtains a time difference.
Specifically, can determine the receiving time of the data packet, determining reception when server receives a data packet
Time is the time on server time axis using server time axis as standard.It, can further according to the object time in data packet
The time difference between the object time in receiving time and data packet to calculate data packet.Before currently processed data packet
In the multiple data packets corresponding time difference, there are a minimum time is poor, which is in multiple time differences
Minimum value, using the minimum value as the mapping value between object time axis terminal time axis corresponding with the data packet.
Wherein, multiple data packets before currently processed data packet can be the institute before currently processed data packet
There is data packet, the partial data packet being also possible to before currently processed data packet.If more before currently processed data packet
A data packet, the partial data packet before being currently processed data packet are multiple in order to improve the convergence of minimum time difference
The selection of data packet can have certain rule, for example, interval is chosen.
The embodiment of the present application is mainly illustrated by taking all data packets before currently processed data packet as an example.With Fig. 5
For the minimum value obtained in multiple time differences in the embodiment of the present application is illustrated.
As shown in figure 5, handling if server is currently received the data packet n in Media Stream A data packet n, obtain
The corresponding mapping value of data packet n.In Media Stream A, all data packets before data packet n are respectively data packet 1 to data packet
N-1, difference between the receiving time TS1 and object time TA1 of data packet 1 are Δ 1=TS1-TA1, when the reception of data packet 2
Between difference between TS2 and object time TA2 be Δ 2=TS2-TA2, until the receiving time TS (n-1) and mesh of data packet n-1
Marking the difference between time TA (n-1) is Δ (n-1)=TS (n-1)-TA (n-1).
Multiple data packets corresponding time difference before then obtaining data packet n is Δ 1, Δ 2 ..., Δ (n-1).It obtains
The minimum value in multiple time differences taken is min (Δ 1, Δ 2 ..., Δ (n-1)), with the min (Δ 1, Δ 2 ..., Δ (n-1))
As the mapping value between object time axis and the time shaft of terminal A.
In this embodiment, step S222 can be executed after step S221.In addition, due in present embodiment,
The corresponding mapping value of data packet is calculated according to the data packet before the data packet in Media Stream where the data packet, therefore,
For speed up processing, the execution opportunity of step S222 is not limited to after step S221, for example, step S222 can also be with
It is performed simultaneously with step S221.
Specifically, server can calculate the number when obtaining data packet and obtaining the acquisition time of the data packet simultaneously
According to where packet in Media Stream, multiple data packets corresponding time difference before the data packet, before obtaining the data packet
Minimum time is poor, as mapping value.
It is illustrated by taking Fig. 5 as an example, server, can be with the corresponding time difference Δ of data packet 1 when getting data packet 1
1 is poor as minimum time；Server is when getting data packet 2, using Δ 1 as mapping value；Server is getting data packet 3
When, the acquisition time of data packet 3 is obtained, and calculate min (Δ 1, Δ 2) simultaneously, mapping value is used as with min (Δ 1, Δ 2)；Service
Device obtains the acquisition time of data packet 4 when getting data packet 4, and calculates min (Δ 1, Δ 2, Δ 3) simultaneously, with min (Δ
1, Δ 2, Δ 3) it is used as mapping value.Server obtains the acquisition time of data packet n when obtaining data packet n, and calculates simultaneously
Min (Δ 1, Δ 2 ..., Δ (n-1)), to calculate mapping of the min (Δ 1, Δ 2 ..., Δ (n-1)) obtained as data packet n
Value.
In addition, optional, when server obtains some data packet corresponding mapping value, with the data packet before the data packet
The minimum time difference of calculating can calculate all data packets before the data packet and the data packet as mapping value simultaneously
The corresponding time difference, it is poor to obtain a minimum time, the mapping value as next data packet.
It is illustrated by taking Fig. 5 as an example, server, can be with the corresponding time difference Δ of data packet 1 when getting data packet 1
1 is poor as minimum time, using the receiving time of data packet 1 as mapping time；Server is when getting data packet 2, with Δ 1
As mapping value, and calculate min (Δ 1, Δ 2)；Server, can be with min (Δ 1, Δ 2) conduct when getting data packet 3
Mapping value, and calculate min (Δ 1, Δ 2, Δ 3).Server is when obtaining data packet n, with min (Δ 1, Δ 2 ..., Δ (n-1))
As mapping value, and calculates min (Δ 1, Δ 2 ..., Δ (n-1), Δ n) is for mapping value as data packet n+1.
In addition, the data packet corresponding time difference may be than the time difference of other data packets when obtaining a data packet
It is smaller, that is to say, that on the Media Stream for including the data packet, the data packet currently obtained the corresponding time difference is likely less than this
Other data packets corresponding time difference before data packet.For the accuracy of calculating, in the embodiment of the present application, provide another
Kind embodiment.In this embodiment, using the time difference of currently processed data packet as one of acquisition minimum time difference
Compare numerical value.That is, in this embodiment, compared to previous embodiment, before step S2227, can also wrap
Include the calculating of the time difference to currently processed data packet, and it is poor with the minimum time that has obtained to compare the time difference, with than
Compared in result, smaller value is used as mapping value.Specifically, referring to Fig. 7, which includes:
Step S2221: for each data packet, obtaining in the Media Stream including the data packet, more before the data packet
A data packet corresponding time difference, obtain multiple time differences.Wherein, each time difference is corresponding data packet at mixed flow end
Receiving time on time shaft and the difference between the object time on terminal time axis.
Step S2224: the minimum value in the multiple time difference is obtained.
For currently processed data packet, where the available data packet in Media Stream, the data before the data packet
Wrap the minimum value in the corresponding time difference.
Step S2225: the data packet corresponding time difference is obtained, as current time difference.
For currently processed data packet, according to its receiving time and object time, can calculate receiving time and
Time difference between object time, defining the time difference is current time difference.
Step S2226: compare whether current time difference is less than the minimum value.If it is not, executing step S2227；If so, holding
Row step S2228.
Step S2227: using the minimum value as the mapping value.
Step S2228: using the current time difference as the mapping value.
If current time difference is greater than or equal to minimum value, when which is that currently processed data packet is corresponding minimum
Between it is poor, can be using it as the mapping value of the data packet.If current time difference is smaller than minimum value, then it represents that current time difference is more received
It holds back in the difference of object time axis and terminal time axis, using current time difference as reflecting between object time axis and terminal time axis
Penetrate value.
Still it is illustrated by taking Fig. 5 as an example.Server can calculate the data packet 1 before data packet n to data packet n-1 points
Not corresponding time difference Δ 1, Δ 2 ..., Δ (n-1), and obtain Δ 1, Δ 2 ..., minimum value min (Δ 1, Δ in Δ (n-1)
2 ..., Δ (n-1)).Server receives data packet n, then can calculate the receiving time TSn and object time TAn of data packet n
Between time difference Δ n=TSn-TAn.Compare the size between Δ n and min (Δ 1, Δ 2 ..., Δ (n-1)).If Δ n is less than
Min (Δ 1, Δ 2 ..., Δ (n-1)) indicates min (Δ 1, Δ 2 ..., Δ (n-1), Δ n)=Δ n, when using Δ n as target
Between mapping value between axis terminal time axis corresponding with data packet n.If Δ n is not less than min (Δ 1, Δ 2 ..., Δ (n-1)),
Indicate min (Δ 1, Δ 2 ..., Δ (n-1), Δ n)=min (Δ 1, Δ 2 ..., Δ (n-1)), with min (Δ 1, Δ 2 ..., Δ
(n-1)) as the mapping value between object time axis terminal time axis corresponding with data packet n.
Optionally, in this embodiment, step S2221 and step S2224 can be executed before step S221 or
Person and step S221 are performed simultaneously.Specifically, for server when obtaining the data packet, obtaining should for any one data packet
Before the acquisition time of data packet or while the acquisition time of the data packet, can by Media Stream where the data packet,
Minimum time difference in other data packets corresponding time difference, which calculates, to be obtained.
Optionally, in this embodiment, for any one data packet, server, can also when obtaining the data packet
After getting the data packet corresponding time difference, the time difference corresponding with other data packets time difference to be compared
Compared with obtaining a minimum time difference as mapping value.That is, in this embodiment, it, can for any one data packet
It include in the Media Stream of the data packet to obtain, the data packet and multiple data packets before the data packet are corresponding
Time difference obtains multiple time differences.The minimum value for obtaining multiple time difference again, using the minimum value as the mapping value.With
, can be after the time difference Δ n for obtaining data packet n for Fig. 5, calculating min, (Δ 1, Δ 2 ..., Δ (n-1), Δ n) are used as number
According to the corresponding mapping value of packet n.
The embodiment of the present application also provides a kind of embodiments.In this embodiment, for currently processed data packet,
The minimum value in multiple data packets corresponding multiple time differences before the data packet obtained, can be previous number
According to the mapping value of packet.From selecting a minimum value as the number in the mapping value of time difference of the data packet and previous data packet
According to the mapping value of packet.That is, in this embodiment, in each Media Stream, first data packet can be with the number
According to the time difference of packet as mapping value.For an any other data packet, the available data packet corresponding time difference, make
For current time difference.Compare the mapping value whether current time difference is less than last data packet.If so, using current time difference as should
The mapping value of data packet；If it is not, using the mapping value of previous data packet as the mapping value of the data packet.
Still it is illustrated with Media Stream shown in fig. 5.Server receives data packet 1, with the corresponding time of data packet 1
Poor Δ 1 is used as the corresponding mapping value of data packet 1, using the receiving time of data packet 1 as mapping time.Server is getting number
When according to wrapping 2, the time difference Δ 2 of data packet 2 is calculated, the size between Δ 2 and the mapping value of data packet 1 is compared, if Δ 2 is less than Δ
1, the mapping value min (Δ 1, Δ 2) using Δ 2 as data packet 2, otherwise mapping value min (Δ 1, the Δ using Δ 1 as data packet 2
2)；Server calculates the time difference Δ 3 of data packet 3, compares the mapping value min of Δ 3 Yu data packet 2 when getting data packet 3
Size between (Δ 1, Δ 2), if Δ 3 is less than min (Δ 1, Δ 2), mapping value min (Δ 1, Δ using Δ 3 as data packet 3
2, Δ 3), the otherwise mapping value min (Δ 1, Δ 2, Δ 3) with min (Δ 1, Δ 2) as data packet 3.Server is getting number
When according to packet n, the time difference Δ n of data packet n is calculated, mapping value min (Δ 1, Δ 2 ..., the Δ (n- of Δ n Yu data packet n-1 are compared
1) size between), if Δ n is less than min (Δ 1, Δ 2 ..., Δ (n-1)), the mapping value min (Δ using Δ n as data packet n
1, Δ 2 ..., Δ (n-1), Δ n), the otherwise mapping value min (Δ with min (Δ 1, Δ 2 ..., Δ (n-1)) as data packet n
1, Δ 2 ..., Δ (n-1), Δ n).
Step S223: according to the corresponding mapping value of each data packet, by the acquisition time of media data in each data packet
It is mapped to the object time axis, obtains the corresponding mapping time of each data packet.
It, can be by the acquisition time of media data in the data packet after obtaining its mapping value for any one data packet
It is mapped to object time axis.Specifically, the acquisition time that can be calculated in the data packet is reflected with corresponding for each data packet
The sum of value is penetrated, the mapping time of the object time axis is mapped to as the acquisition time in the data packet.
By taking Fig. 5 as an example, if server receives data packet n, and the mapping value for calculating acquisition data packet n is min (Δ 1, Δ
2, Δ 3 ... Δ n), the then mapping time that the acquisition time t ' An in data packet n is mapped to object time axis are equal to t ' An+min
(Δ 1, Δ 2, Δ 3 ... Δ n).
Step S230: according to the sequencing of the mapping time of each data packet, mixed flow is carried out to the multipath media stream.
In the time for being mapped to the object time axis according to multipath media stream, mixed flow is carried out to the multipath media stream
When, mixed flow can be carried out according to the sequencing of the mapping time of data packet in each road Media Stream.
Wherein, when mixed flow, the mapping time of data packet in each road Media Stream can be aligned according to preset mixed flow rule,
The data packet of alignment is mixed.
Wherein it is possible to by each road Media Stream, the identical packet synchronisation alignment of mapping time.Such as, server receives
Two-way Media Stream, respectively Media Stream A and Media Stream B, if the mapping time of the data packet n in Media Stream A is the m moment,
The data packet that mapping time in Media Stream B is the m moment is aligned with data packet n.
If the mapping time of some data packet of certain road Media Stream is not identical as any data packet of other Media Streams,
Using the data in the data packet as the data of the mapping time.Such as, server receives Media Stream A and Media Stream B, media
The mapping time for flowing the data packet n in A is the m moment, and not having mapping time in Media Stream B is the data packet at m moment, at the m moment
Only have the data in data packet n in mixed data.
Alternatively, it is also possible to preset the unit time section of mixed flow, the data packet in each unit time section is aligned.Example
Such as, using t as the length in unit time section, server initially receives two-way Media Stream, respectively Media Stream A from the m moment
And Media Stream B, then by the packet combining in the two-way Media Stream in m to m+t time interval；By m+t to m+2t time zone
Interior packet combining, by the packet combining etc. in m+2t to m+3t time interval.The unit time section it is specific
Value does not limit, such as can be the data collection cycle etc. of terminal.
When data in the data packet by alignment are mixed, including by picture mix and sound mix, Ke Yigen
It is mixed according to actual demand.For example, when sound mix, it can be by the sampled value phase of the audio data of the phase in data packet in the same time
Add；It, can be according to display demand mixing when picture mixes.
Such as demand is shown as shown in figure 8, wherein showing that picture A indicates the picture of terminal end A, corresponding Media Stream A；Show picture
Face B indicates the picture of terminal B, corresponding Media Stream B.Display picture B is less than display picture A and is covered in display picture A, shows picture
Face A is (x0, y0) in top left co-ordinate, and bottom right angular coordinate is (xn, yn)；The top left co-ordinate for showing picture B is (xm, y0),
Bottom right angular coordinate is (xk, yk).It, can be by Media Stream A when the data packet being then aligned in mixed-media stream A and Media Stream B
Data packet in video frame be placed in lowest level, it is (x0, y0) that displaing coordinate, which is set as top left co-ordinate, and bottom right angular coordinate is
(xn, yn)；Video frame in the data packet of Media Stream B is placed in upper layer, top left co-ordinate is (xm, y0), and bottom right angular coordinate is
(xk, yk).
For another example, demand is shown as shown in figure 9, wherein showing that picture A indicates the picture of terminal end A, corresponding Media Stream A；It is aobvious
Show that picture B indicates the picture of terminal B, corresponding Media Stream B.Display picture A is (x0, y0), bottom right angular coordinate in top left co-ordinate
For (x (n/2), yn)；The top left co-ordinate for showing picture B is (x (n/2), y0), and bottom right angular coordinate is (xn, yn).Then mixing
When the data packet being aligned in Media Stream A and Media Stream B, upper left can be set by the video frame in the data packet of Media Stream A
Angular coordinate is (x0, y0), and bottom right angular coordinate is (x (n/2), yn)；By the video frame top left co-ordinate in the data packet of Media Stream B
It is set as (x (n/2), y0), bottom right angular coordinate is (xn, yn).
Wherein, show that the coordinate of picture is not necessarily coordinate in the display screen for receiving the mixed terminal singly flowed,
Actual coordinate in display screen can be determined according to the display interval position of display screen and display interval size.
In the embodiment of the present application, above-mentioned alignment and hybrid mode by way of example only, specific alignment and hybrid mode
It does not limit, can determine according to actual needs.
In the embodiment of the present application, if mixed flow end is terminal, terminal is when receiving multipath media stream, if multipath media stream
In include Media Stream that local Media Stream and other terminals are sent, can be using the time shaft of local Media Stream described in
The Media Stream that other terminals are sent is mapped to the object time axis by object time axis.The Media Stream that other terminals are sent reflects
The mode for being mapped to the object time axis may refer to foregoing manner, and details are not described herein.
In addition, can also be obtained by test in order to which mixed flow alignment is more accurate from the terminal of transmission Media Stream to arrival
Close to a minimal time delay of ideal time delay between mixed flow end.If mixed flow end be terminal, terminal when receiving multipath media stream,
If including the Media Stream that local Media Stream and other terminals are sent in multipath media stream, the time shaft of local Media Stream can
To be the time shaft of the terminal plus the minimal time delay.Or when obtaining terminal log again according to from the ideal process for collecting transmission
Between, when the time shaft that the time shaft of local Media Stream can be the terminal adds the ideal process plus the minimal time delay
Between.The mode that the Media Stream that other terminals are sent is mapped to the object time axis may refer to foregoing manner, no longer superfluous herein
It states.If mixed flow end is server, the mapping of the Media Stream of server can be reached using server time axis as object time axis
Value can subtract the minimal time delay, or subtract the minimal time delay again on the basis of preceding method calculates the mapping value obtained
Subtract the ideal process time.
In the embodiment of the present application, each road media stream to mixed flow end, in Media Stream, the receiving time and data of data packet
The minimum value of time difference will gradually restrain between the object time in packet, and gradually converge on object time Zhou Yugai road Media Stream
The difference of terminal time axis.Therefore, for each data packet in every road Media Stream, the minimum time that can currently will be obtained
Acquisition time in the data packet is mapped to by difference as the mapping value between server time axis and corresponding terminal time axis
Object time axis obtains the mapping time that acquisition time is mapped to object time axis.The mapping of data packet on each road Media Stream
Time is all the time on object time axis, so that the time standard for being aligned each data packet is consistent, when making mixed flow
The alignment synchronization of data packet is more accurate, improves synchronous effect.
For example, for two-way Media Stream, the time shaft of the terminal of first via Media Stream 1 minute faster than object time axis, the
The time shaft of two road Media Streams 1 minute slower than object time axis, in this two-way Media Stream simultaneously object time axis when
Between the lower 2 points of acquisitions in 15 minutes of standard audio, video data formed data packet, then in the first via Media Stream data packet carry adopt
Collect the time be 2 points 16 minutes, the acquisition time that the data packet carries in the second road Media Stream be 2 points 14 minutes.By the two-way Media Stream
After being mapped to object time axis, then acquisition time is 2 points of 16 minutes data packets in first via Media Stream, and mapping time is 2: 15
Point；Then in the second road Media Stream acquisition time be 2 points of 14 minutes data packets, mapping time be 2 points 15 minutes.It is real then in mixed flow
Border is will be 2 under object time axis in lower 2 points of 15 minutes the second road of the data mixing Media Streams of object time axis in first via Media Stream
15 minutes data of point.
The embodiment of the present application also provides a kind of flow mixing devices 300.Referring to Figure 10, which includes: receiving module
310, for receiving multipath media stream；Mapping block 320, for being respectively mapped to the acquisition time of the multipath media stream together
One object time axis obtains the mapping time of corresponding different media flows；And mixed flow module 330, for according to the multichannel matchmaker
Body stream is mapped to the mapping time that the object time axis obtains, and carries out mixed flow to the multipath media stream.
Wherein, every road Media Stream includes multiple data packets, and mapping block 320 can be used for matchmaker in each data packet respectively
The acquisition time of volume data is mapped to the object time axis, obtains the corresponding mapping time of each data packet, when the acquisition
Between for time on corresponding terminal time axis, the mapping time is that the acquisition time of media data in each data packet reflects
It is mapped to the time of the object time axis.Mixed flow module 330 can be used for according to the successive suitable of the mapping time of each data packet
Sequence carries out mixed flow to the multipath media stream.
Optionally, mapping block 320 may include: time acquisition unit, for obtaining media data in each data packet
Acquisition time；Mapping value acquiring unit is used for for each data packet, when obtaining the object time axis with corresponding terminal
Between mapping value between axis；Map unit is used for according to the corresponding mapping value of each data packet, by each Package media data
Acquisition time be mapped to the object time axis, obtain the corresponding mapping time of each data packet.
Optionally, mapping value acquiring unit may include: that the first difference obtains subelement, include the data packet for obtaining
Media Stream in, multiple data packets corresponding time difference before the data packet, obtain multiple time differences, wherein each
Time difference is receiving time of the corresponding data packet on the time shaft of mixed flow end and between the object time on terminal time axis
Difference；First minimum value obtains subelement, for obtaining the minimum value in the multiple time difference；First mapping subelement,
For using the minimum value as the mapping value.
Optionally, for each data packet, the first difference obtain subelement can be also used for obtaining the data packet it is corresponding
Time difference, as current time difference.Mapping value acquiring unit can also include comparing subunit, be for comparing current time difference
It is no to be less than the minimum value.If comparison result be it is yes, first mapping subelement can be used for using the current time difference as institute
State mapping value；If comparison result be it is no, first mapping subelement can be used for using the minimum value as the mapping value.
Optionally, mapping value acquiring unit may include that the second difference obtains subelement, include the data packet for obtaining
Media Stream in, the data packet and multiple data packets corresponding time difference before the data packet, when obtaining multiple
Between it is poor, wherein each time difference be receiving time of the corresponding data packet on the time shaft of mixed flow end on terminal time axis
Object time between difference；Second minimum value obtains subelement, for obtaining the minimum value of the multiple time difference；Second
Subelement is mapped, for using the minimum value as the mapping value.
Optionally, in the embodiment of the present application, for each data packet, mapping block 320 can be used for calculating the data
The sum of the acquisition time of media data and corresponding mapping value, are mapped to the mesh as the object time in the data packet in packet
Mark the mapping time of time shaft.
It is apparent to those skilled in the art that for convenience and simplicity of description, above-mentioned each method
It can be cross-referenced between embodiment；The specific work process of foregoing description device and module can be implemented with reference to preceding method
Corresponding process in example, details are not described herein.
In several embodiments provided herein, the mutual coupling of module can be electrical property, mechanical or other
The coupling of form.
It, can also be in addition, can integrate in a processing module in each functional module in each embodiment of the application
It is that modules physically exist alone, can also be integrated in two or more modules in a module.Above-mentioned integrated mould
Block both can take the form of hardware realization, can also be realized in the form of software function module.
Figure 11 is please referred to, it illustrates the structural block diagrams of a kind of electronic equipment 400 provided by the embodiments of the present application.The electronics
Equipment 400 can be terminal or server.The electronic equipment one or more processors 410 (one is only shown in figure), are deposited
Reservoir 420 and one or more programs.Wherein, one or more of programs are stored in the memory 420, and by
It is configured to be executed by one or more of processors 410.One or more of programs are configured to carry out previous embodiment
Described method.
Processor 410 may include one or more processing core.Processor 410 is whole using various interfaces and connection
Various pieces in a electronic equipment 400, by run or execute the instruction being stored in memory 420, program, code set or
Instruction set, and the data being stored in memory 420 are called, execute the various functions and processing data of electronic equipment 400.It can
Selection of land, processor 410 can use Digital Signal Processing (Digital Signal Processing, DSP), field-programmable
Gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic
Array, PLA) at least one of example, in hardware realize.Processor 410 can integrating central processor (Central
Processing Unit, CPU), in image processor (Graphics Processing Unit, GPU) and modem etc.
One or more of combinations.Wherein, the main processing operation system of CPU, user interface and application program etc.；GPU is for being responsible for
Show the rendering and drafting of content；Modem is for handling wireless communication.It is understood that above-mentioned modem
It can not be integrated into processor 410, be realized separately through one piece of communication chip.
Memory 420 may include random access memory (Random Access Memory, RAM), also may include read-only
Memory (Read-Only Memory).Memory 420 can be used for store instruction, program, code, code set or instruction set.It deposits
Reservoir 420 may include storing program area and storage data area, wherein the finger that storing program area can store for realizing operating system
Enable, for realizing the instruction of at least one function, for realizing instruction of above-mentioned each embodiment of the method etc..Storage data area is also
It can be with data (such as phone directory, audio, video data, chat record data) etc. that electronic equipment is created in use.
In addition, the electronic equipment can also include display screen, for carrying out video if the electronic equipment 400 is terminal
Display；The electronic equipment can also include audio-frequence player device, for carrying out audio broadcasting；The electronic equipment can also include wheat
Gram wind, for carrying out audio collection；.
Figure 12 is please referred to, it illustrates a kind of structural frames of computer readable storage medium provided by the embodiments of the present application
Figure.Program code is stored in the computer readable storage medium 500, said program code can call execution above-mentioned by processor
Method described in embodiment of the method.
Computer readable storage medium 500 can be such as flash memory, EEPROM (electrically erasable programmable read-only memory),
The electronic memory of EPROM, hard disk or ROM etc.Optionally, computer readable storage medium 500 includes non-volatile meter
Calculation machine readable medium (non-transitory computer-readable storage medium).Computer-readable storage
Medium 500 has the memory space for the program code 510 for executing any method and step in the above method.These program codes can
With from reading or be written in one or more computer program product in this one or more computer program product.
Program code 510 can for example be compressed in a suitable form.
Finally, it should be noted that above embodiments are only to illustrate the technical solution of the application, rather than its limitations；Although
The application is described in detail with reference to the foregoing embodiments, those skilled in the art are when understanding: it still can be with
It modifies the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；And
These are modified or replaceed, do not drive corresponding technical solution essence be detached from each embodiment technical solution of the application spirit and
Range.
Claims (11)
1. a kind of mixed flow method, which is characterized in that the mixed flow end applied to Media Stream, which comprises
Receive multipath media stream；
The acquisition time of the multipath media stream is mapped to same object time axis respectively, obtains reflecting for corresponding different media flows
Penetrate the time；
It is mapped to the mapping time that the object time axis obtains according to the multipath media stream, the multipath media stream is carried out
Mixed flow.
2. described respectively the method according to claim 1, wherein every road Media Stream includes multiple data packets
The acquisition time of the multipath media stream is mapped to same object time axis, obtains the mapping time of corresponding different media flows, root
It is mapped to the mapping time that the object time axis obtains according to the multipath media stream, mixed flow is carried out to the multipath media stream,
Include:
The acquisition time of media data in each data packet is mapped to the object time axis respectively, obtains each data packet pair
The mapping time answered, the acquisition time are the time on corresponding terminal time axis, and the mapping time is each data
The acquisition time of media data is mapped to the time of the object time axis in packet；
According to the sequencing of the mapping time of each data packet, mixed flow is carried out to the multipath media stream.
3. described according to the method described in claim 2, it is characterized in that, received multipath media stream is sent by different terminals
The acquisition time of media data in each data packet is mapped to same object time axis respectively, it is corresponding to obtain each data packet
Mapping time, comprising:
Obtain the acquisition time of media data in each data packet；
For each data packet, the mapping value between the object time axis and corresponding terminal time axis is obtained；
According to the corresponding mapping value of each data packet, the acquisition time of media data in each data packet is mapped to the target
Time shaft obtains the corresponding mapping time of each data packet.
4. according to the method described in claim 3, obtaining the object time it is characterized in that, described for each data packet
Mapping value between axis and corresponding terminal time axis, comprising:
For each data packet, obtaining includes in the Media Stream of the data packet, and multiple data packets before the data packet are right respectively
The time difference answered obtains multiple time differences, wherein each time difference is reception of the corresponding data packet on the time shaft of mixed flow end
Time and the difference between the object time on terminal time axis；
Obtain the minimum value in the multiple time difference；
Using the minimum value as the mapping value.
5. according to the method described in claim 4, it is characterized in that, it is described using the minimum value as the mapping value before,
Further include:
The data packet corresponding time difference is obtained, as current time difference；
Compare whether current time difference is less than the minimum value；
If so, using the current time difference as the mapping value,
If it is not, using the minimum value as the mapping value.
6. according to the method described in claim 3, obtaining the object time it is characterized in that, described for each data packet
Mapping value between axis and corresponding terminal time axis, comprising:
For each data packet, obtains and include in the Media Stream of the data packet, the data packet and more before the data packet
A data packet corresponding time difference, obtain multiple time differences, wherein each time difference is corresponding data packet at mixed flow end
Receiving time on time shaft and the difference between the object time on terminal time axis；
Obtain the minimum value of the multiple time difference；
Using the minimum value as the mapping value.
7. according to the described in any item methods of claim 4-6, which is characterized in that the object time is matchmaker in the data packet
The acquisition time of volume data or the sending time of data packet.
8. according to the method described in claim 3, it is characterized in that, according to the corresponding mapping value of each data packet, by every number
It is mapped to the object time axis according to the acquisition time of media data in packet, when obtaining the corresponding mapping of each data packet
Between, comprising:
For each data packet, the sum of acquisition time of media data and corresponding mapping value in the data packet are calculated, as this
The acquisition time of media data is mapped to the mapping time of the object time axis in data packet.
9. a kind of flow mixing device, which is characterized in that described device includes:
Receiving module, for receiving multipath media stream；
Mapping block is corresponded to for the acquisition time stream of the multi-path media to be mapped to same object time axis respectively
The mapping time of different media flows；
Mixed flow module, for being mapped to the mapping time that the object time axis obtains according to the multipath media stream, to described
Multipath media stream carries out mixed flow.
10. a kind of electronic equipment characterized by comprising
One or more processors；
Memory；
One or more programs, wherein one or more of programs are stored in the memory and are configured as by described
One or more processors execute, and one or more of programs are configured to carry out as claim 1-8 is described in any item
Method.
11. a kind of computer readable storage medium, which is characterized in that be stored with program generation in the computer readable storage medium
Code, said program code can be called by processor and execute the method according to claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811526963.XA CN109769124A (en) | 2018-12-13 | 2018-12-13 | Mixed flow method, apparatus, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811526963.XA CN109769124A (en) | 2018-12-13 | 2018-12-13 | Mixed flow method, apparatus, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109769124A true CN109769124A (en) | 2019-05-17 |
Family
ID=66451819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811526963.XA Pending CN109769124A (en) | 2018-12-13 | 2018-12-13 | Mixed flow method, apparatus, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109769124A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110650353A (en) * | 2019-09-25 | 2020-01-03 | 广州华多网络科技有限公司 | Multi-person continuous-wheat mixed drawing method and device, storage medium and electronic equipment |
| CN110933450A (en) * | 2019-10-16 | 2020-03-27 | 咪咕文化科技有限公司 | Multi-channel live broadcast synchronization method, system, edge device, terminal and storage medium |
| CN113473162A (en) * | 2021-04-06 | 2021-10-01 | 北京沃东天骏信息技术有限公司 | Method, device and equipment for playing media stream and computer storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102752669A (en) * | 2011-04-19 | 2012-10-24 | 中国电信股份有限公司 | Transfer processing method and system for multi-channel real-time streaming media file and receiving device |
| US20130243205A1 (en) * | 2010-05-04 | 2013-09-19 | Shazam Entertainment Ltd. | Methods and Systems for Disambiguation of an Identification of a Sample of a Media Stream |
| CN103702013A (en) * | 2013-11-28 | 2014-04-02 | 北京航空航天大学 | Frame synchronization method for multiple channels of real-time videos |
| CN103814338A (en) * | 2011-07-20 | 2014-05-21 | 航空网络公司 | Systems and methods of network synchronization |
| WO2015158368A1 (en) * | 2014-04-15 | 2015-10-22 | Telefonaktiebolaget L M Ericsson (Publ) | Synchronised social tv |
| CN105491393A (en) * | 2015-12-02 | 2016-04-13 | 北京暴风科技股份有限公司 | Method for implementing multi-user live video business |
| CN108206966A (en) * | 2016-12-16 | 2018-06-26 | 杭州海康威视数字技术股份有限公司 | A kind of video file synchronous broadcast method and device |
-
2018
- 2018-12-13 CN CN201811526963.XA patent/CN109769124A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130243205A1 (en) * | 2010-05-04 | 2013-09-19 | Shazam Entertainment Ltd. | Methods and Systems for Disambiguation of an Identification of a Sample of a Media Stream |
| CN102752669A (en) * | 2011-04-19 | 2012-10-24 | 中国电信股份有限公司 | Transfer processing method and system for multi-channel real-time streaming media file and receiving device |
| CN103814338A (en) * | 2011-07-20 | 2014-05-21 | 航空网络公司 | Systems and methods of network synchronization |
| CN103702013A (en) * | 2013-11-28 | 2014-04-02 | 北京航空航天大学 | Frame synchronization method for multiple channels of real-time videos |
| WO2015158368A1 (en) * | 2014-04-15 | 2015-10-22 | Telefonaktiebolaget L M Ericsson (Publ) | Synchronised social tv |
| CN105491393A (en) * | 2015-12-02 | 2016-04-13 | 北京暴风科技股份有限公司 | Method for implementing multi-user live video business |
| CN108206966A (en) * | 2016-12-16 | 2018-06-26 | 杭州海康威视数字技术股份有限公司 | A kind of video file synchronous broadcast method and device |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110650353A (en) * | 2019-09-25 | 2020-01-03 | 广州华多网络科技有限公司 | Multi-person continuous-wheat mixed drawing method and device, storage medium and electronic equipment |
| CN110650353B (en) * | 2019-09-25 | 2020-12-04 | 广州华多网络科技有限公司 | Multi-person continuous-wheat mixed drawing method and device, storage medium and electronic equipment |
| CN110933450A (en) * | 2019-10-16 | 2020-03-27 | 咪咕文化科技有限公司 | Multi-channel live broadcast synchronization method, system, edge device, terminal and storage medium |
| CN113473162A (en) * | 2021-04-06 | 2021-10-01 | 北京沃东天骏信息技术有限公司 | Method, device and equipment for playing media stream and computer storage medium |
| CN113473162B (en) * | 2021-04-06 | 2023-11-03 | 北京沃东天骏信息技术有限公司 | Media stream playing method, device, equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP4287634A1 (en) | Game data processing method and apparatus, and storage medium | |
| CN103379363B (en) | Method for processing video frequency and device, mobile terminal and system | |
| CN106993239B (en) | Information display method in live broadcast process | |
| CN109769124A (en) | Mixed flow method, apparatus, electronic equipment and storage medium | |
| US11826643B2 (en) | Reducing latency in cloud gaming applications by overlapping reception and decoding of video frames and their display | |
| CN108462883B (en) | A kind of living broadcast interactive method, apparatus, terminal device and storage medium | |
| CN111654720B (en) | Video encoding method, apparatus, device and readable storage medium | |
| CN113209632B (en) | Cloud game processing method, device, equipment and storage medium | |
| CN112272327B (en) | Data processing method, device, storage medium and equipment | |
| CN110933450B (en) | Multi-channel live broadcast synchronization method, system, edge device, terminal and storage medium | |
| CN102752669A (en) | Transfer processing method and system for multi-channel real-time streaming media file and receiving device | |
| CN109218755A (en) | A kind for the treatment of method and apparatus of media data | |
| CN108833809A (en) | A kind of video mixed flow control method, device, system, equipment and medium | |
| CN108881894A (en) | The multimedia Quality of experience of VR determines method and device | |
| CN106303661A (en) | A kind of live client realizes the method and system of self adaptation screen rotation | |
| US20220401835A1 (en) | Overlapping scan-in and encode at the server | |
| CN104219571A (en) | Method and device for automatically providing watching focus | |
| CN109413371A (en) | Video frame rate calculation method and device | |
| CN114828972A (en) | High-speed scanout of server display buffers for cloud gaming applications | |
| CN112954380B (en) | Video playing processing method and device | |
| CN114786022A (en) | New media live broadcast and user online interaction method | |
| CN100471209C (en) | High quality manu picture service device of mobile communication terminal | |
| CN112261422A (en) | Simulation remote live broadcast stream data processing method suitable for broadcasting and television field | |
| CN117041628B (en) | Live picture rendering method, system, device, equipment and medium | |
| CN109274955A (en) | The compression of light field video and depth map and synchronous method, system and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210113 Address after: 511442 3108, 79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Applicant after: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Address before: 511400 24th floor, building B-1, North District, Wanda Commercial Plaza, Wanbo business district, No.79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou, Guangdong Province Applicant before: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20190517 Assignee: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. Assignor: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Contract record no.: X2021440000054 Denomination of invention: Mixed flow method, device, electronic equipment and storage medium License type: Common License Record date: 20210208 |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190517 |