CN105429983B - Acquire method, media termination and the music lesson system of media data - Google Patents
Acquire method, media termination and the music lesson system of media data Download PDFInfo
- Publication number
- CN105429983B CN105429983B CN201510846324.1A CN201510846324A CN105429983B CN 105429983 B CN105429983 B CN 105429983B CN 201510846324 A CN201510846324 A CN 201510846324A CN 105429983 B CN105429983 B CN 105429983B
- Authority
- CN
- China
- Prior art keywords
- frame
- audio
- sent
- buffering area
- buffer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 239000000872 buffer Substances 0.000 claims abstract description 110
- 230000003139 buffering effect Effects 0.000 claims abstract description 87
- 230000005540 biological transmission Effects 0.000 claims abstract description 69
- 238000012546 transfer Methods 0.000 claims abstract description 6
- 238000001514 detection method Methods 0.000 claims description 16
- 239000000284 extract Substances 0.000 abstract description 4
- 230000006835 compression Effects 0.000 description 6
- 238000007906 compression Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 241001269238 Data Species 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B15/00—Teaching music
- G09B15/001—Boards or like means for providing an indication of chords
- G09B15/002—Electrically operated systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4335—Housekeeping operations, e.g. prioritizing content for deletion because of storage space restrictions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4392—Processing of audio elementary streams involving audio buffer management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/4508—Management of client data or end-user data
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses method, media termination and the music lesson systems of acquisition media data.Media termination includes video acquisition unit, screen buffer, audio collection unit, audio buffer, sends buffering area, transmission unit and control unit.Video acquisition unit acquires image and is encoded to video frame.Screen buffer is suitable for storage video frame.Audio collection unit acquires sound and is encoded to audio frame.Audio buffer is suitable for storage audio frame.It sends buffering area and stores data frame to be sent.Each data frame is video frame or audio frame.Transmission unit is suitable for data frame transfer to be sent to media server.Control unit is adapted to detect for audio buffer and is pushed to audio frame therein to send in buffering area.When audio buffer is empty, if the quantity of data frame to be sent is less than threshold value and screen buffer is not sky, control unit extracts video frame from screen buffer and is pushed to transmission buffering area.
Description
Technical field
The present invention relates to the communications fields, more particularly to method, media termination and the music lesson system of acquisition media data.
Background technology
Currently, in the real-time Communication for Power scheme such as such as video conference or network direct broadcasting, the terminal for acquiring media data can be with
Acquire the media datas such as video frame and audio frame, and to play end transmission media data.Such as it can be by video frame and audio frame
Open network after encapsulating together is carried out to transmit.Alternatively, video frame and audio frame are respectively packaged and are transmitted by terminal.
However, the problems such as existing Network status is complicated and changeable, and for example there is network jitter, intermittent interruptions.Acquisition
Terminal by network when sending audio and video data, the problems such as there are network delay, network congestions.Therefore, media play
When end obtains audio and video data from acquisition terminal, the disconnected slack problem of card is faced.
Invention content
For this purpose, the present invention provides a kind of scheme of new acquisition media data, effective solution is at least one above to ask
Topic.
According to an aspect of the present invention, a kind of media termination, including video acquisition unit, screen buffer, sound are provided
Frequency collecting unit, sends buffering area, transmission unit and control unit at audio buffer.Video acquisition unit is suitable for acquisition image
And it is encoded to video frame.Screen buffer is suitable for video frame of the storage from video acquisition unit.Audio collection unit is suitable for adopting
Collection sound is simultaneously encoded to audio frame.Audio buffer is suitable for audio frame of the storage from audio collection unit.It is suitable to send buffering area
In the one or more data frames to be sent of storage.Wherein each data frame to be sent be video frame from screen buffer or
Audio frame from audio buffer.Transmission unit is suitable for one or more data frame transfers to be sent to media services
Device.Control unit is adapted to detect for audio buffer and is pushed to audio frame therein to send in buffering area, and in Audio Buffer
When area is empty, judge whether that the quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not sky.
If the quantity of data frame to be sent is less than threshold value and screen buffer is not sky, control unit is extracted from screen buffer
One frame video frame is simultaneously pushed in transmission buffering area.
According to a further aspect of the invention, a kind of method of acquisition media data is provided.This method was suitable at media end
It is executed in end.The media termination includes sending buffering area.It sends buffering area and is suitable for storing one or more data frames to be sent.
Wherein each data frame to be sent is video frame or audio frame.This approach includes the following steps.It acquires image and is encoded to and regard
Frequency frame, and the video frame is stored in screen buffer.Acquisition sound is simultaneously encoded to audio frame, and the audio frame is stored sound
In frequency buffering area.Audio frame therein is simultaneously pushed in transmission buffering area by detection audio buffer.If audio buffer is
Sky then judges whether that the quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not sky.If
The quantity of data frame to be sent is less than threshold value and screen buffer is not sky, then extracts a frame from the screen buffer and regard
Frequency frame is simultaneously pushed in transmission buffering area.By one or more of data frame transfers to be sent to media server.
According to a further aspect of the invention, a kind of music lesson system is provided, including media termination according to the present invention,
Media server and media play end.Media server is suitable for receiving the audio frame and video frame that media termination is sent.Media are broadcast
End is put to be suitable for obtaining from media server and playing audio frame and video frame.
It is slow preferentially can be pushed to transmission by medium data acquisition scheme according to the present invention for audio buffer sound intermediate frequency frame
Area is rushed, and when audio buffer is that quantity that is empty and sending data frame in buffering area is not above threshold value, it just can be by video
Video frame is pushed to transmission buffering area in buffering area.In this way, medium data acquisition scheme according to the present invention can be preferential always
Audio frame is transmitted, and video frame is transmitted in the gap of transmission audio frame.In particular, when network bandwidth is relatively low
(quantity for sending data frame in buffering area is more than threshold value), medium data acquisition scheme according to the present invention can stop to hair
Buffering area pushing video frame is sent, and normally audio buffer sound intermediate frequency frame is pushed to and is sent in buffering area.Such work
Mode can effectively reduce transmission unit when network bandwidth is relatively low and need the data volume sent, to effectively ensure audio frame
Real-time Transmission.In other words, medium data acquisition scheme according to the present invention realizes the prioritised transmission to audio frame, to keep away
The disconnected problem of card of the played sound in media play end is exempted from.It should be noted that in the sound such as music teaching occasion of high importance
In, medium data acquisition scheme according to the present invention ensures the real-time Transmission of voice data, and media play end can play
Continuously, distortionless audio data, to which the Experience Degree of user be greatly improved.
Description of the drawings
To the accomplishment of the foregoing and related purposes, certain illustrative sides are described herein in conjunction with following description and drawings
Face, these aspects indicate the various modes that can put into practice principles disclosed herein, and all aspects and its equivalent aspect
It is intended to fall in the range of theme claimed.Read following detailed description in conjunction with the accompanying drawings, the disclosure it is above-mentioned
And other purposes, feature and advantage will be apparent.Throughout the disclosure, identical reference numeral generally refers to identical
Component or element.
Fig. 1 shows the block diagram according to an example of the present invention music lesson system 100；
Fig. 2 shows the block diagrams of media termination 200 according to some embodiments of the invention；And
Fig. 3 shows the flow chart of the method 300 of acquisition media data according to some embodiments of the present invention.
Specific implementation mode
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
Completely it is communicated to those skilled in the art.
Fig. 1 shows the block diagram according to an example of the present invention music lesson system 100.As shown in Figure 1, music teaching
System 100 may include multiple student clients 110, server 120 and teacher's client 130.In music lesson system 100
In, student client 110 and teacher's client 130 carry out real-time Communication for Power by server 120, to carry out Online Music religion
It learns.For example, when student plays, student client 110 may be implemented as media termination, and acquisition student plays relevant
The media datas such as video and audio, and these media datas are transmitted to teacher's client 130 by server 120.Teacher
Client 130 may be implemented as media play end, receive and play media data, so that teacher understands the performance of student in real time
Situation.Meanwhile teacher's client 130 can also be implemented as media termination, feedback guidance that acquisition teacher plays student or
The media data of the contents such as person's teaching demonstration, and transmitted to student client by server 120.Student client 110 can be with
Be implemented as media play end, receive and play the media data from teacher's client 130, so as to teacher to student play into
Row Real-time Feedback, or teaching demonstration is carried out to student in real time.In short, student client 110 and teacher's client 130 all may be used
To be implemented as media termination and media broadcast terminal.Here, media data such as fingering, breath, pleasure including playing an instrument
Device sound and the contents of courses such as official documents and correspondence are instructed, but not limited to this.
In general, music lesson system 100 faces in the unstable problem of transmission bandwidth, such as network jitter, network interval
Situations such as disconnected.However, the music teaching of high quality have to the real-time, synchronism and fluency etc. of media data it is higher
It is required that.The present invention is directed to the acquisition media data link of music lesson system, it is proposed that a kind of new media termination.With reference to
Fig. 2 carries out further exemplary illustration to the media termination in music lesson system.Certainly, media termination can be student visitor
Family end or teacher's client do not repartition the concrete type of media termination hereinafter to simplify the description.Equally, Xue Shengke
Family end 110 and teacher's client 130 may be implemented as media play end.It should be noted that media according to the present invention
Terminal can be used in music lesson system, and but it is not limited to this.For example, media termination according to the present invention can also answer
In the real time flow mediums schemes such as such as video conference, match live streaming.
Fig. 2 shows the block diagrams of media termination 200 according to some embodiments of the invention.Here, computing device can be by
It is configured to media termination 200.And computing device can be implemented as a part for portable (or mobile) electronic equipment of small size, this
A little electronic equipments can be that such as cellular phone, personal digital assistant (PDA), personal media player device, wireless network are clear
Look at equipment, personal helmet, application specific equipment or may include any of the above function mixing apparatus.Computing device
It is also implemented as including desktop computer and the personal computer of notebook computer configuration, but not limited to this.
As shown in Fig. 2, media termination 200 includes video acquisition unit 210, screen buffer 220, audio collection unit
230, audio buffer 240, transmission buffering area 250, transmission unit 260 and control unit 270.
Video acquisition unit 210 is suitable for acquisition image and is encoded to video frame.For example, video acquisition unit 210 is to student
The scene to play an instrument is shot, to obtain sequence of frames of video.According to an embodiment of the present invention, video acquisition unit 210
Including camera 211 and coding unit 212.Camera 211 is suitable for acquisition original image frame.The acquisition parameter example of original image frame
It is such as 640*480 sizes and 25 frames/second, but not limited to this.In addition, video acquisition unit 210 can also be former in one frame of every capture
When beginning picture frame, current time value is recorded as the first time stamp for capturing the original image frame.According to a reality of the invention
Example is applied, the format sample of original image frame is：
{dwstamp videodata}
Wherein, dwstamp is to stab at the first time, and videodata is the picture frame of YUV420 formats.
Coding unit 212 is suitable for encoding original image frame.For example, coding unit 212 can be to original image frame
H.264, progress (is constituted jointly by ITU-T Video Coding Experts Groups (VCEG) and ISO/IEC dynamic image expert groups (MPEG)
Joint video team (JVT, Joint Video Team) propose high compression digital video coding-coding device standard) format volume
Code.The coding parameter of coding unit 212 includes complete picture group (GOP).The length of GOP such as 100 frames, i.e. an image group tool
There are 100 frame video frame.Here, GOP is started with I frames, with followed by several P frames.Also there is B frames in adjacent P interframe.For example,
The partial sequence of one GOP is I P B P B P P P P B P.Wherein, I frames are intracoded frame, contain complete image
Information, and carrying out reconstruction image without reference to any additional information.P frames be forward-predictive-coded frames, by before it P frames or
Person's I frames are predicted.B frames are bi-directional predicted condensed frame.When picture frame is compressed into B frames, coding unit 212 is according to adjacent
Former frame, this frame and latter frame data difference carry out distilled edition frame.The coding parameter of coding unit 212 can also include
Frame per second and coding size are encoded, but not limited to this.According to an embodiment of the present invention, coding unit 212 executes encoding operation
Example code is as follows：
Ret=CLDC_Open (width, height, bitrate, mode)；
width:The width of video compress
height:The height of video compress
bitrate:The target bit rate of video compress
mode:Video compressed mode controls
// compression of images engine initializes
Ret=CLDC_Encode (pBuf, dwbase, m_pVideoBuffer, nEncoderLen)；
pBuf:The image data buffer of acquisition
dwbase:The length of raw image data
m_pVideoBuffer：The buffering of compression of images
nEncoderLen：Length after compression of images
The H264 image code streams for including SPS and PPS are generated after // image coding
Ret=CLDC_Close ()；
// image encodes engine and closes
Screen buffer 220 is suitable for storing the video frame that video acquisition unit 210 is generated.Here, screen buffer 220
For example, buffer circle.For example, screen buffer 220 can store 20 frame data being newly generated always.
Audio collection unit 230 is suitable for acquisition sound and is encoded to audio frame.In an embodiment in accordance with the invention, sound
Frequency collecting unit 230 can carry out ACC (Advanced Audio Coding, Advanced Audio Coding) lattice to the sound acquired
The coding of formula, generation code check are 192kbps.Here, music when audio collection unit 230 can acquire instrument playing is gentle
Breath, but not limited to this.In an embodiment in accordance with the invention, audio collection unit 230 carries out audio collection and encoding operation
Correlative code example it is as follows：
Int nSamples=AUDIO_SAMPLERATE；// audio sample rate
Int nChannels=1；// channel number
Int nBits=16；The digit of // audio sample
Int nAudioBitrate=192000；The bit rate of // audio output
Int nRet=0；
NRet=m_pFaacCodec->Open(nSamples,nChannels,nBits,nAudioBitrate)；
// setting audio compression parameter, and open audio compressor
NCodecRet=m_pFaacCodec->Encode(pBuf,nLen,streamabuffer,nEncodeLen)；
pBuf:Original audio data
nLen:The length of audio data
streamabuffer:Encode the buffering of output
nEncodeLen:Data length after coding
// audio coding is carried out, the data after coding are [ADTS]+[audio compression data]
NRet=m_pFaacCodec->Close()；
// audio coding terminates
In addition, audio collection unit 230 can also capture the time value of acquired audio frame, and it was recorded as the second time
Stamp.Each corresponding second timestamp of audio frame is, for example, the acquisition time of first audio sample point of the frame.
Audio buffer 240 is suitable for storing the audio frame that audio collection unit 230 is generated.Here, audio buffer 240
For example, buffer circle.Can also include the second corresponding timestamp per frame audio frame.In this way, media play end is being obtained
It, can be according to first time stamp and the second timestamp to audio frame when getting the audio frame and video frame of the transmission of media termination 200
Processing is synchronized with video frame.
It sends buffering area 250 and is suitable for storing one or more data frames to be sent.Wherein each data frame to be sent
For the video frame from screen buffer 220 or the audio frame from audio buffer 240.
Transmission unit 260 is suitable for send data frame to be sent in buffering area 250 and is transferred to media server successively.This
Sample, media server can transmit data frame to be sent to media play end.Media play end receives and playing video data
And audio data.
As described above, sending storage in buffering area 250 has the data frame for waiting for transmission unit 260 to transmit.Control unit 270
Buffering area 250 is sent suitable for extracting data frame from audio buffer 220 and screen buffer 240 and being pushed to.In general,
Such as in the stream media systems such as music teaching, the importance of sound is higher than the data such as video information.Control unit 270 is each
When executing push operation, audio buffer 220 is detected into row major.A kind of situation is that audio buffer 240 has audio
Frame, control unit 270 extract a frame audio frame and are pushed to transmission buffering area 250.Another situation is that control unit 270 detects
It is sky to audio buffer 240.In other words, audio buffer 240 does not need when control unit 270 executes this push operation
The audio frame of transmission.Control unit 270 continues to detect whether video frame buffers 220 are empty and detection is sent in buffering area 250
Whether the quantity of data frame to be sent is more than threshold value.Threshold value is, for example, 5.Here, why to detect and send in buffering area 250
Whether data frame is more than threshold value, is in order to by sending whether number of data frames in buffering area 250 judges current transmission unit 260
In blocked state (i.e. Network Abnormal).In other words, transmission unit 260 will can be sent in time when being not in blocked state
Data frame carries out network transmission in buffering area 250.So, the data frame for having more than threshold value will not be accumulated by sending buffering area 250.Instead
It, transmission unit 260 will cannot send data frame in buffering area 250 and carry out network transmission in time when in blocked state.Hair
It can be more than threshold value to send the quantity of the data frame accumulated in buffering area 250.It should be noted that the present invention is to detecting screen buffer
220 sequences that buffering area 250 is sent with detection do not do excessive restriction.In one embodiment, control unit 270 detects regard first
Whether frequency buffering area 220 is empty, and no longer detection sends data frame data in buffering area 250 when screen buffer 220 is empty
Whether superthreshold.In other words, if screen buffer 220 is sky, terminate this push operation.Here, adjacent push twice
The time interval of operation is, for example, 10 milliseconds.In addition, if screen buffer 220 is not sky, the detection of control unit 270 sends slow
Rush in area 250 data frame data amount whether superthreshold.In yet another embodiment, control unit 270, which first detects, sends buffering area
250, and when number of data frames is less than threshold value, whether detection screen buffer 220 is empty.According to one implementation of the present invention
In, the example code of 270 course of work of control unit is as follows：
As described above, control unit 270 is detecting screen buffer 220 not and is empty and to be sent number of data frames not
When more than threshold value, extracts a frame video frame from screen buffer 220 and be pushed to transmission buffering area 250.To sum up, according to the present invention
Control unit 270 preferentially 240 sound intermediate frequency frame of audio buffer can be pushed to send buffering area 250, and in Audio Buffer
It, just can be by video in screen buffer 220 when area 240 is that quantity that is empty and sending data frame in buffering area is not above threshold value
Frame, which is pushed to, sends buffering area 250.In this way, media termination 200 according to the present invention can always preferentially pass audio frame
It is defeated, and video frame is transmitted in the gap of transmission audio frame.In particular, (sending buffering area when network bandwidth is relatively low
The quantity of data frame is more than threshold value in 250), media termination 200 according to the present invention can stop pushing to transmission buffering area 250
Video frame, and normally 240 sound intermediate frequency frame of audio buffer is pushed to and is sent in buffering area 250.Such working method can
The data volume sent is needed to effectively reduce transmission unit 260 when network bandwidth is relatively low, to effectively ensure audio frame
Real-time Transmission.In other words, media termination 200 according to the present invention realizes the prioritised transmission to audio frame, so as to avoid matchmaker
Body plays the disconnected problem of card of the played sound in end.It should be noted that in the sound such as music teaching occasion of high importance, according to
The present invention media termination ensure voice data real-time Transmission, and then media play end can play play it is continuous, undistorted
Audio data.
In addition, when the quantity of data frame in sending buffering area 250 is more than threshold value, control unit 270 confirms current network
It is abnormal.Control unit 270 is further adapted for calculating transmission code check of the transmission unit 260 to sending video frame in buffering area, so as to video
Collecting unit 210 is adjusted the generation code check of video frame according to the transmission code check.According to an embodiment of the present invention, it controls
Unit 270 can count in the predetermined time (such as 2 seconds), transmission unit to sending the quantity forwarded of video frame in buffering area, into
And it calculates and sends code check.Then, video acquisition unit 210 can according to send code check to image acquisition parameter and coding parameter into
Row is adjusted, to adjust the generation code check of video frame.For example, video acquisition unit 210 can be to the size of acquisition original image
(resolution ratio) or acquisition frame rate are adjusted.In another example video acquisition unit 220 can be to encoding original image frame
Coding parameter when operation is adjusted.Coding parameter is such as including encoding frame per second, generating the coding size of video frame.This
Sample, video acquisition unit 210 is after adjusting generation code check so that generates transmission code of the code check with transmission unit 260 to video frame
Rate matches.Further, transmission unit 260 can send video frame in real time, and effectively reduces and send buffering area 250
Middle number of data frames be more than threshold value the case where occur, and avoid due in screen buffer 220 video frame rate it is excessively high and
The problem of cannot being transmitted in time.In addition, since coding frame per second can be adjusted in video acquisition unit so that transmission unit
The sequence of 260 video frame transmitted is uniform on acquisition time.In this way, media play end avoids played video
The problem of frame delay is excessive and picture jumps.
Fig. 3 shows the flow chart of the method 300 of acquisition media data according to some embodiments of the invention.Method 300
Suitable for being executed in media termination according to the present invention.
As shown in figure 3, method 300 starts from step S310.In step S310, acquires sound and be encoded to audio frame, and
The audio frame is stored in audio buffer.Method 300 can also include step S320, acquire image and be encoded to video
Frame, and the video frame is stored in screen buffer.Here, audio buffer and screen buffer are, for example, loop buffer.
In addition, in step s 320, the current time for capturing the corresponding original image of video frame can also be recorded, as at the first time
Stamp.Each video frame can also include stabbing corresponding first time.In step S310, when can also record the acquisition of audio frame
Between, as the second timestamp.Here the second timestamp is, for example, the timestamp of first sampled point of audio frame.Audio frame can be with
Including the second timestamp.Media play end is when getting the audio frame and video frame of the transmission of media termination 200 in this way, Ke Yigen
Processing is synchronized to audio frame and video frame according to first time stamp and the second timestamp.
Media termination according to the present invention includes sending buffering area.It is one or more to be sent suitable for storage to send buffering area
Data frame.Each data frame to be sent is the video frame from screen buffer or the audio frame from audio buffer.
For the audio frame being stored in audio buffer and the video frame being stored in screen buffer, method 300 is by executing step
Rapid S330, S340 and S350 control the network transmission of audio frame and video frame.In step S330, Audio Buffer is detected
Audio frame therein is simultaneously pushed in transmission buffering area by area.In addition, if detecting that audio buffer is in step S330
Sky, then method 300 execute step S340.In step S340, judge whether the quantity for sending data frame to be sent in buffering area
It is less than threshold value and screen buffer is not sky.Specifically, according to one embodiment of present invention, it is first in step S340
First whether detection screen buffer is empty.If screen buffer is sky, currently without the video frame that needs are sent, method
300 continue to execute step S330.If video frame is not sky, continue the number that detection sends data frame to be sent in buffering area
Whether amount is more than threshold value.If number of data frames to be sent is more than threshold value, then illustrate current transmission network congestion.In order to preferential
Audio frame is transmitted, then video frame will not be pushed to transmission buffering area, but continue to execute step S330.It is another according to the present invention
A embodiment, in step S340, whether the quantity for detecting data frame to be sent in currently transmitted buffering area first is more than threshold
Value.If it exceeds the threshold, then no longer detecting screen buffer, but continue to execute step S330.
In addition, if in step S340, detect that the quantity of data frame to be sent is less than threshold value, and video is slow
It is sky to rush area not, thens follow the steps S350.In step S350, a frame video frame is extracted from screen buffer and is pushed to hair
Send buffering area.
As described above, according to the method for the present invention 300 be suitable for by execute step S310 and S320 come generate audio frame and
Video frame, and the audio frame and video frame to be sent are selected by executing step S330, S340 and S350, in turn
Data frame to be sent is stored in and is sent in buffering area.For sending data frame in buffering area, method 300 is by executing step
S360 carries out network transmission.In step S360, data frame transfer to be sent in buffering area will be sent to media server.
300 more specific embodiments are consistent with the working method of media termination in Fig. 2 200 according to the method for the present invention, here no longer
It repeats.
A10, the method as described in A8 or A9, wherein the screen buffer, the audio buffer and described send are delayed
It is loop buffer to rush area.A11, the method as described in any one of A8-A10, wherein described to judge whether to wait in transmission buffering area
The quantity of transmission data frame is less than threshold value and screen buffer is not that empty step includes：Detection screen buffer whether be
Sky, if being not sky, whether the quantity for continuing to detect data frame to be sent in the transmission buffering area is more than threshold value.A12, such as
Method described in any one of A8-A11, wherein described to judge whether that the quantity for sending data frame to be sent in buffering area does not surpass
It crosses threshold value and screen buffer is not that empty step includes：Whether detect the quantity for sending data frame to be sent in buffering area
More than threshold value, and when being less than threshold value, whether detection screen buffer is empty.A13, the side as described in any one of A8-A12
Method, wherein the video frame includes stabbing at the first time, which is the capture time of the corresponding image of the video frame
Value；The audio frame includes the second timestamp, which is the capture time value of the corresponding sound of the audio frame.A14、
Method as described in any one of A8-A13, wherein the threshold value is 5.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that the implementation of the present invention
Example can be put into practice without these specific details.In some instances, well known method, knot is not been shown in detail
Structure and technology, so as not to obscure the understanding of this description.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of each inventive aspect,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：It is i.e. required to protect
Shield the present invention claims the feature more features than being expressly recited in each claim.More precisely, as following
As claims reflect, inventive aspect is all features less than single embodiment disclosed above.Therefore, it abides by
Thus the claims for following specific implementation mode are expressly incorporated in the specific implementation mode, wherein each claim itself
As a separate embodiment of the present invention.
Those skilled in the art should understand that the module of the equipment in example disclosed herein or unit or groups
Part can be arranged in equipment as depicted in this embodiment, or alternatively can be positioned at and the equipment in the example
In different one or more equipment.Module in aforementioned exemplary can be combined into a module or be segmented into addition multiple
Submodule.
Those skilled in the art, which are appreciated that, to carry out adaptively the module in the equipment in embodiment
Change and they are arranged in the one or more equipment different from the embodiment.It can be the module or list in embodiment
Member or component be combined into a module or unit or component, and can be divided into addition multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it may be used any
Combination is disclosed to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so to appoint
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification (including adjoint power
Profit requires, abstract and attached drawing) disclosed in each feature can be by providing the alternative features of identical, equivalent or similar purpose come generation
It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included certain features rather than other feature, but the combination of the feature of different embodiments means in of the invention
Within the scope of and form different embodiments.For example, in the following claims, embodiment claimed is appointed
One of meaning mode can use in any combination.
In addition, be described as herein can be by the processor of computer system or by executing for some in the embodiment
The combination of method or method element that other devices of the function are implemented.Therefore, have for implementing the method or method
The processor of the necessary instruction of element forms the device for implementing this method or method element.In addition, device embodiment
Element described in this is the example of following device：The device is used to implement performed by the element by the purpose in order to implement the invention
Function.
As used in this, unless specifically stated, come using ordinal number " first ", " second ", " third " etc.
Description plain objects are merely representative of the different instances for being related to similar object, and are not intended to imply that the object being described in this way must
Must have the time it is upper, spatially, in terms of sequence or given sequence in any other manner.
Although the embodiment according to limited quantity describes the present invention, above description, the art are benefited from
It is interior it is clear for the skilled person that in the scope of the present invention thus described, it can be envisaged that other embodiments.Additionally, it should be noted that
The language that is used in this specification primarily to readable and introduction purpose and select, rather than in order to explain or limit
Determine subject of the present invention and selects.Therefore, without departing from the scope and spirit of the appended claims, for this
Many modifications and changes will be apparent from for the those of ordinary skill of technical field.For the scope of the present invention, to this
The done disclosure of invention is illustrative and not restrictive, and it is intended that the scope of the present invention be defined by the claims appended hereto.
Claims (13)
1. a kind of media termination, including：
Video acquisition unit suitable for acquisition image and is encoded to video frame；
Screen buffer is suitable for video frame of the storage from the video acquisition unit；
Audio collection unit suitable for acquisition sound and is encoded to audio frame；
Audio buffer is suitable for audio frame of the storage from the audio collection unit；
Buffering area is sent, is suitable for storing one or more data frames to be sent, wherein each data frame to be sent is from institute
State the video frame of screen buffer or the audio frame from the audio buffer；
Transmission unit is suitable for one or more of data frame transfers to be sent to media server；And
Control unit is adapted to detect for audio buffer and is pushed to audio frame therein to send in buffering area, and slow in audio
When to rush area be empty, judge whether that the quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not
Sky,
If the quantity of data frame to be sent is less than threshold value and screen buffer is not sky, extracted from the screen buffer
One frame video frame is simultaneously pushed in transmission buffering area；
Wherein, described control unit is further adapted for when the quantity of data frame to be sent in detecting transmission buffering area is more than threshold value,
Transmission code check of the transmission unit to video frame is detected, and code check adjusting parameter is generated according to the transmission code check；And
The video acquisition unit is further adapted for adjusting the generation code check of video frame according to the code check adjusting parameter.
2. media termination as described in claim 1, wherein the screen buffer, the audio buffer and the transmission
Buffering area is loop buffer.
3. media termination as claimed in claim 1 or 2, wherein described control unit is suitable for being judged whether according to following manner
The quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not sky：
Whether detection screen buffer is empty, if being not sky, continues to detect data frame to be sent in the transmission buffering area
Whether quantity is more than threshold value.
4. media termination as claimed in claim 1 or 2, wherein described control unit is suitable for being judged whether according to following manner
The quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not sky：
Whether the quantity for detecting data frame to be sent in the transmission buffering area is more than threshold value, and when being less than threshold value, detection
Whether screen buffer is empty.
5. media termination as claimed in claim 1 or 2, wherein
The video frame includes stabbing at the first time, which is the capture time value of the corresponding image of the video frame；
The audio frame includes the second timestamp, which is the capture time value of the corresponding sound of the audio frame.
6. media termination as claimed in claim 1 or 2, wherein the threshold value is 5.
7. a kind of method of acquisition media data, suitable for being executed in media termination, which includes sending buffering area, is fitted
In the one or more data frames to be sent of storage, wherein each data frame to be sent is video frame or audio frame, this method
Including：
Acquisition image is simultaneously encoded to video frame, and the video frame is stored in screen buffer；
Acquisition sound is simultaneously encoded to audio frame, and the audio frame is stored in audio buffer；
Audio frame therein is simultaneously pushed in transmission buffering area by detection audio buffer；
If audio buffer is sky, judge whether that the quantity for sending data frame to be sent in buffering area is less than threshold value and regards
Frequency buffering area is not sky,
If the quantity of data frame to be sent is less than threshold value and screen buffer is not sky, carried from the screen buffer
It takes a frame video frame and is pushed to and send in buffering area；
When the quantity of data frame to be sent in detecting transmission buffering area is more than threshold value, detection transmission unit is to video frame
Code check is sent, and code check adjusting parameter is generated according to the transmission code check；
The generation code check of video frame is adjusted according to the code check adjusting parameter；And
By one or more of data frame transfers to be sent to media server.
8. the method for claim 7, wherein the screen buffer, the audio buffer and transmission buffering
Area is loop buffer.
9. method as claimed in claim 7 or 8, wherein the number for judging whether to send data frame to be sent in buffering area
Amount is less than threshold value and screen buffer is not that empty step includes：
Whether detection screen buffer is empty, if being not sky, continues to detect data frame to be sent in the transmission buffering area
Whether quantity is more than threshold value.
10. method as claimed in claim 7 or 8, wherein the number for judging whether to send data frame to be sent in buffering area
Amount is less than threshold value and screen buffer is not that empty step includes：
Whether the quantity for detecting data frame to be sent in the transmission buffering area is more than threshold value, and when being less than threshold value, detection
Whether screen buffer is empty.
11. method as claimed in claim 7 or 8, wherein
The video frame includes stabbing at the first time, which is the capture time value of the corresponding image of the video frame；
The audio frame includes the second timestamp, which is the capture time value of the corresponding sound of the audio frame.
12. method as claimed in claim 7 or 8, wherein the threshold value is 5.
13. a kind of music lesson system, including：
Media termination as described in any one of claim 1-6；
Media server is suitable for receiving audio frame and video frame that the media termination is sent；And
Media play end, suitable for being obtained from the media server and playing the audio frame and video frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510846324.1A CN105429983B (en) | 2015-11-27 | 2015-11-27 | Acquire method, media termination and the music lesson system of media data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510846324.1A CN105429983B (en) | 2015-11-27 | 2015-11-27 | Acquire method, media termination and the music lesson system of media data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105429983A CN105429983A (en) | 2016-03-23 |
| CN105429983B true CN105429983B (en) | 2018-09-14 |
Family
ID=55507925
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510846324.1A Active CN105429983B (en) | 2015-11-27 | 2015-11-27 | Acquire method, media termination and the music lesson system of media data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105429983B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106791575B (en) * | 2016-12-22 | 2019-11-15 | 上海寰视网络科技有限公司 | A kind of control method and equipment that data are sent |
| CN107566769B (en) * | 2017-09-27 | 2019-12-03 | 维沃移动通信有限公司 | A kind of video recording method and mobile terminal |
| CN110535810A (en) * | 2018-05-25 | 2019-12-03 | 视联动力信息技术股份有限公司 | A kind of processing method and terminal of video data |
| CN109361945A (en) * | 2018-10-18 | 2019-02-19 | 广州市保伦电子有限公司 | The meeting audiovisual system and its control method of a kind of quick transmission and synchronization |
| CN111182220A (en) * | 2020-01-09 | 2020-05-19 | 西安万像电子科技有限公司 | Image processing apparatus, remote device, and communication system |
| CN111585684B (en) * | 2020-05-14 | 2021-08-10 | 武汉大学 | Multi-path camera time alignment method and system for networked monitoring video analysis |
| CN112689160B (en) * | 2020-11-27 | 2022-12-09 | 烟台艾睿光电科技有限公司 | Video transmission method and device applied to image acquisition equipment |
| CN113131972B (en) * | 2021-04-13 | 2022-04-15 | Oppo广东移动通信有限公司 | Method, device, electronic equipment and storage medium for receiving and transmitting audio data packet |
| CN113905266B (en) * | 2021-10-11 | 2024-04-30 | 杭州萤石软件有限公司 | Audio and video data processing method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1481547A (en) * | 2000-12-15 | 2004-03-10 | ���˹���Ѷ��� | Encoding audio signals |
| CN1744592A (en) * | 2005-09-29 | 2006-03-08 | 西安交通大学 | Multi-source flow media transmission QoS control method |
| CN101030938A (en) * | 2007-02-05 | 2007-09-05 | 北京大学 | System and method for organizing and transmitting QoS self-adaptive P2P flow medium data |
| CN101340590A (en) * | 2008-08-14 | 2009-01-07 | 北京创毅视讯科技有限公司 | Multiplex apparatus and multiplex method |
| CN101479686A (en) * | 2006-06-27 | 2009-07-08 | 皇家飞利浦电子股份有限公司 | A device and a method for managing power consumption of a plurality of data processing units |
| CN102223516A (en) * | 2010-04-14 | 2011-10-19 | 奥多比公司 | Media quality enhancement among connected media communication devices |
-
2015
- 2015-11-27 CN CN201510846324.1A patent/CN105429983B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1481547A (en) * | 2000-12-15 | 2004-03-10 | ���˹���Ѷ��� | Encoding audio signals |
| CN1744592A (en) * | 2005-09-29 | 2006-03-08 | 西安交通大学 | Multi-source flow media transmission QoS control method |
| CN101479686A (en) * | 2006-06-27 | 2009-07-08 | 皇家飞利浦电子股份有限公司 | A device and a method for managing power consumption of a plurality of data processing units |
| CN101030938A (en) * | 2007-02-05 | 2007-09-05 | 北京大学 | System and method for organizing and transmitting QoS self-adaptive P2P flow medium data |
| CN101340590A (en) * | 2008-08-14 | 2009-01-07 | 北京创毅视讯科技有限公司 | Multiplex apparatus and multiplex method |
| CN102223516A (en) * | 2010-04-14 | 2011-10-19 | 奥多比公司 | Media quality enhancement among connected media communication devices |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105429983A (en) | 2016-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105429983B (en) | Acquire method, media termination and the music lesson system of media data | |
| WO2016150317A1 (en) | Method, apparatus and system for synthesizing live video | |
| KR101453239B1 (en) | Streaming encoded video data | |
| KR100800716B1 (en) | Apparatus and method for tranmitting and receiving moving picture data using a close range communication | |
| CN105429984B (en) | Media playing method, equipment and music lesson system | |
| CN1812587A (en) | Video coding | |
| CN105516090B (en) | Media playing method, equipment and music lesson system | |
| CN105430453B (en) | Medium data acquisition method, media termination and Online Music tutoring system | |
| CN105072506B (en) | The transmission method and system of live teaching broadcast video | |
| CN106162199B (en) | Method and system for video processing with back channel message management | |
| KR101065819B1 (en) | Video encoding apparatus and video encoding method | |
| EP3792731A1 (en) | Multimedia information transmission method and apparatus, and terminal | |
| CN112565224B (en) | Video processing method and device | |
| CN109040818B (en) | Audio and video synchronization method, storage medium, electronic equipment and system during live broadcasting | |
| EP2214413A2 (en) | Server and client selective video frame pathways | |
| JP6319970B2 (en) | Image communication apparatus and control method thereof | |
| JP6193569B2 (en) | RECEPTION DEVICE, RECEPTION METHOD, AND PROGRAM, IMAGING DEVICE, IMAGING METHOD, AND PROGRAM, TRANSMISSION DEVICE, TRANSMISSION METHOD, AND PROGRAM | |
| Hoppe et al. | Video Streaming over IP using the DASH Technique | |
| JP2010011287A (en) | Image transmission method and terminal device | |
| CN108124183A (en) | With it is synchronous obtain it is audio-visual to carry out the method for one-to-many video stream | |
| JP6400163B2 (en) | Reception device, reception method, transmission device, transmission method, and program | |
| JP5799958B2 (en) | Video processing server and video processing method | |
| JP2007324722A (en) | Moving picture data distribution apparatus and moving picture data communication system | |
| KR102184131B1 (en) | Multi channels transmitting system for dynamaic audio and controlling method | |
| JP5036081B2 (en) | Image transmission system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |