CN113315931B - HLS stream-based data processing method and electronic equipment - Google Patents
HLS stream-based data processing method and electronic equipment Download PDFInfo
- Publication number
- CN113315931B CN113315931B CN202110765617.2A CN202110765617A CN113315931B CN 113315931 B CN113315931 B CN 113315931B CN 202110765617 A CN202110765617 A CN 202110765617A CN 113315931 B CN113315931 B CN 113315931B
- Authority
- CN
- China
- Prior art keywords
- subtitle
- video frame
- webvtt
- stream
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/278—Subtitling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
Abstract
The application discloses a data processing method based on HLS flow and an electronic device, wherein the method comprises the following steps: acquiring TS slices and WebVTT subtitle slices of HLS streams; converting the TS slices to obtain video frames; converting the WebVTT subtitle slices to obtain intermediate subtitle data; embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying captions; and obtaining and outputting TS stream according to the embedded video frame. The method and the device realize conversion display of the WebVTT subtitle on the existing set top box or television equipment, so that the video with the WebVTT subtitle on the Internet can be displayed and played on the existing set top box or television equipment.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data processing method and an electronic device based on HLS streams.
Background
With the rapid development of the current internet era, the method has important significance for the development of the economic society and the promotion of the informatization degree, and meanwhile, the internet continuously changes the life style and the working style of human beings and provides more convenience, so that all industries in the society draw close to the internet.
There are many streaming media protocols commonly used in the internet, and formats of subtitle files carried by the protocols are also various, for example, WebVTT subtitle files are generally carried by streaming media data of the HLS protocol. In the existing set-top box or television equipment, due to the limitation of decoding capability, received TS stream audio and video data in a specific format can only be added when a video is produced or embedded into the audio and video data in a specific form of subtitles if subtitles are to be displayed.
Therefore, the existing set-top box or television equipment cannot support the WebVTT subtitle file, so that the existing network set-top box or television equipment cannot display the subtitle carried by the streaming media data when playing the streaming media data of the HLS protocol, and the trend of the traditional set-top box or television equipment merging into the internet is hindered.
Disclosure of Invention
The data processing method and the electronic device based on the HLS stream can convert the WebVTT subtitle and embed the WebVTT subtitle into a video frame, and solve the problem that the conventional set top box or television device cannot receive and display the WebVTT subtitle.
The application discloses a data processing method based on HLS flow, which comprises the following steps:
acquiring TS slices and WebVTT subtitle slices of HLS streams;
converting the TS slices to obtain video frames;
converting the WebVTT subtitle slices to obtain intermediate subtitle data;
embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying captions;
and obtaining and outputting TS stream according to the embedded video frame.
In an embodiment, after the step of obtaining and outputting a TS stream according to the embedded video frame, the method further includes:
and decoding the TS stream to obtain decoded data carrying subtitles.
In an embodiment, the embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying captions includes:
acquiring time information of the video frame and the intermediate caption data;
obtaining a time difference value according to the time information of the video frame and the intermediate caption data;
if the time difference is smaller than a preset difference, embedding the intermediate caption data into the video frame;
and if the time difference is greater than a preset difference, discarding the intermediate caption data, constructing empty caption data, and embedding the empty caption data into the video frame.
In an embodiment, the acquiring a TS slice and a WebVTT subtitle slice of an HLS stream includes:
acquiring a main play list according to the HLS stream;
extracting an m3u8 file of the WebVTT subtitle from the main playlist;
and acquiring the WebVTT subtitle slice according to the m3u8 file of the WebVTT subtitle.
In an embodiment, the converting the WebVTT subtitle slice to obtain intermediate subtitle data includes:
reading the subtitle content information of the WebVTT subtitle slice to obtain WebVTT subtitle data;
and converting the WebVTT subtitle data based on a preset embedded subtitle standard to obtain the intermediate subtitle data.
In an embodiment, the converting the WebVTT subtitle data based on the preset embedded subtitle standard includes:
acquiring WebVTT subtitle control characters and WebVTT subtitle information from the WebVTT subtitle data;
converting the WebVTT subtitle control characters and the WebVTT subtitle information according to a preset embedded subtitle standard to form intermediate subtitle control characters and intermediate subtitle information;
and placing the intermediate caption information into an intermediate caption queue for extraction.
In an embodiment, the obtaining and outputting a TS stream according to the embedded video frame includes:
and performing code rate output control according to the relation between the system local timestamp and the DTS of the embedded video frame to obtain the TS stream.
In an embodiment, the performing rate output control according to the relationship between the system local timestamp and the DTS of the embedded video frame includes:
obtaining a phase difference value according to the DTS of the current embedded video frame and the local timestamp of the system;
obtaining a sum value according to the difference value and the DTS of the next frame of the embedded video frame;
if the added value is larger than the system local timestamp, outputting the current embedded video frame;
and if the added value is smaller than the system local timestamp, waiting for the added value to be equal to the system local timestamp and then outputting the current embedded video frame.
In an embodiment, the converting the TS slice to obtain a video frame includes:
reading video content information of the TS slices and placing the video content information in a video frame queue;
selecting a complete GOP group video frame from the video frame queue and placing the complete GOP group video frame in a GOP group queue;
and extracting GOP group video frames from the GOP group queue and sequencing according to the PTS of the GOP group video frames.
The application also discloses an electronic device, the electronic device includes:
a processor; and
a memory having stored therein a computer program, the processor being configured to execute the HLS stream-based data processing method by calling the computer program stored in the memory.
As can be seen from the above, in the HLS stream-based data processing method and the electronic device of the present application, the method obtains the intermediate subtitle data by converting the WebVTT subtitle slice, embeds the intermediate subtitle data into the corresponding video frame to form the embedded video frame with the subtitle, converts the embedded video frame to obtain the TS stream, and outputs the TS stream to the existing set top box or television device to display the subtitle, so that the conversion display of the WebVTT subtitle in the existing set top box or television device is realized, and the subtitle compatibility of the existing set top box or television device for the video of the HLS stream is improved.
Drawings
Fig. 1 is a flowchart illustrating an implementation of a data processing method based on HLS streams according to an embodiment of the present disclosure;
fig. 2 is a system framework diagram for executing a data processing method based on HLS streams according to an embodiment of the present application;
fig. 3 is a flowchart illustrating an implementation of embedding intermediate subtitle data into a corresponding video frame to form an embedded video frame with subtitles according to an embodiment of the present application;
fig. 4 is a flowchart illustrating an implementation of obtaining a TS slice and a WebVTT subtitle slice of an HLS stream according to an embodiment of the present application;
fig. 5 is a flowchart illustrating an implementation of rate output control according to a relationship between a system local timestamp and a DTS of an embedded video frame according to an embodiment of the present application;
fig. 6 is a hardware framework diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following detailed description of the preferred embodiments of the present application, taken in conjunction with the accompanying drawings, will make the advantages and features of the present application more readily appreciated by those skilled in the art, and thus will more clearly define the scope of the invention.
The application discloses a data processing method based on HLS streams, which can be operated on electronic equipment such as a set top box or television equipment, and therefore the electronic equipment operated by the method is not limited in the application.
Referring to fig. 1, an implementation flow of a data processing method based on HLS streams according to an embodiment of the present application is shown.
As shown in fig. 1, the data processing method based on HLS stream includes:
101. and acquiring TS slices and WebVTT subtitle slices of the HLS stream.
Specifically, the HLS stream is a Streaming media data stream based on an HLS protocol, and the HLS protocol (HTTP Live Streaming) is a Streaming media network transmission protocol based on HTTP proposed by apple inc. Wherein, TS slice and WebVTT subtitle slice are extracted from the main play list of HLS stream.
102. And converting the TS slices to obtain video frames.
Specifically, the TS slice includes video time information and video content information, the video content information is subjected to frame reading to obtain video frames, and the video frames are placed in a video frame queue for extraction, the video time information is converted to form a PTS of the video frames, the PTS is a display timestamp and includes a start display time and an end display time, and the PTS of the video frames is used to provide a time reference for the video frames, thereby facilitating related operations such as sorting, comparison output, comparison embedding, and the like.
103. And converting the WebVTT subtitle slices to obtain intermediate subtitle data.
Specifically, the intermediate caption data plays an intermediate conversion role, and the intermediate caption data supports receiving and displaying on an existing set top box or television equipment, for example, the intermediate caption data may be embedded caption data, which includes a series of embedded caption data such as EIA-708 standard caption data and EIA-608 standard caption data, and therefore, the intermediate caption data commonly used in the art may be used according to actual situations, and the type of the intermediate caption data specifically used is not limited in the embodiment of the present application.
The WebVTT subtitle slice comprises subtitle time information and subtitle content information, the subtitle content information of the WebVTT subtitle slice is read to obtain WebVTT subtitle data, the WebVTT subtitle data is placed in a WebVTT subtitle queue for extraction, the subtitle time information is converted to form a PTS of the WebVTT subtitle data, the PTS is a display time stamp and comprises display starting time and display ending time, and the PTS of the WebVTT subtitle data is used for providing a time reference for the WebVTT subtitle data, so that related operations such as sorting, comparison output and comparison embedding are facilitated. And then, extracting WebVTT caption data from the WebVTT caption queue, converting the WebVTT caption data to obtain intermediate caption data, placing the intermediate caption data in the intermediate caption queue for extraction, and forming a PTS of the intermediate caption data by converting or transmitting the PTS of the WebVTT caption data.
104. And embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying the caption.
Specifically, the intermediate caption data is extracted from the intermediate caption queue, and the video frame is acquired from the video frame queue. And the intermediate caption data and the video frames are embedded into each corresponding video frame one by one according to the relation between the PTS of the intermediate caption data and the PTS of the video frames to form an embedded video frame carrying the caption.
The intermediate caption data is specifically embedded in the NAL area of the video frame, where the NAL, i.e. the network abstraction layer, is responsible for formatting the data and providing header information to ensure that the data is suitable for transmission on various channels and storage media. A friendly interface is provided for video encoders and transmission systems so that encoded video data can be efficiently transmitted in a variety of different network environments.
105. And obtaining and outputting the TS stream according to the embedded video frame.
Specifically, the embedded video frame may obtain and output a TS stream in an output mode of a video frame, for example, the embedded video frame may obtain and output the TS stream in a CBR mode, that is, obtain and output the TS stream at a constant bitrate, so that a common output mode of a video frame in the art may be adopted according to an actual situation, and the type of the specifically adopted output mode of the video frame is not limited in the embodiment of the present application.
The method also comprises the steps that audio frames are formed, the video frames and the audio frames are combined to form audio and video frames, and the audio frames are transmitted along with the video frames without any processing in the method.
Referring to fig. 2, fig. 2 shows a system framework for running the HLS-stream-based data processing method according to the embodiment of the present application.
As shown in fig. 2, in order to further understand the above technical solution, this embodiment provides a system for operating the method, including a video frame processing module 1, a WebVTT subtitle conversion module 2, a subtitle embedding module 3, and an output control module 4, where the video frame processing module 1 obtains a TS slice, converts the TS slice into a video frame, the WebVTT subtitle conversion module 2 obtains a WebVTT subtitle slice, converts the WebVTT subtitle slice into intermediate subtitle data, the subtitle embedding module 3 receives the video frame and the intermediate subtitle data, embeds the intermediate subtitle data into the corresponding video frame to form an embedded video frame with subtitles, and the output control module 4 obtains a TS stream according to the embedded video frame and outputs the TS stream.
As can be seen from the above, in the HLS stream-based data processing method of the present application, the intermediate subtitle data is obtained by converting the WebVTT subtitle slice, the intermediate subtitle data is embedded into the corresponding video frame to form the embedded video frame carrying the subtitle, the embedded video frame is converted to obtain the TS stream, and the TS stream is output to the existing set top box or television device to display the subtitle, so that the conversion display of the WebVTT subtitle in the existing set top box or television device is realized, and the subtitle compatibility of the existing set top box or television device for the video of the HLS stream is improved.
In an embodiment, after the step of obtaining and outputting the TS stream according to the embedded video frame, the method further includes: and decoding the TS stream to obtain decoded data carrying subtitles.
Specifically, the subtitles carried by the decoded data may be Closed Caption subtitles, which is a commonly used subtitle supporting display of television equipment, and the Closed Caption subtitles are subtitles based on an embedded subtitle standard, and include a series of embedded subtitle standards such as an EIA-708 standard and an EIA-608 standard, for example, after a TS stream is decoded, a set of decoded data completely carrying Closed Caption subtitles is formed and then can be transmitted to an existing set-top box or television equipment for receiving and displaying, so that subtitles commonly used in the field and supporting display of television equipment can be used according to actual situations, and the type of the subtitles displayed by the specifically used supporting television equipment is not limited in the embodiment of the present application.
Referring to fig. 2 again, the system further includes a decoding output module 5, after the TS stream is received by the decoding output module 5, the decoding output module 5 decodes the TS stream to obtain decoded data carrying subtitles, and after the decoded data is obtained by the existing set top box or television equipment, the subtitles can be displayed after being processed.
Referring to fig. 3, fig. 3 shows an implementation process of embedding intermediate subtitle data into a corresponding video frame to form an embedded video frame carrying subtitles according to an embodiment of the present application.
As shown in fig. 3, the embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying captions includes:
201. time information of the video frame and the middle subtitle data is acquired.
The time information of the video frame is the PTS of the video frame, and the time information of the intermediate caption data is the PTS of the intermediate caption data.
202. And obtaining a time difference value according to the time information of the video frame and the middle caption data.
Specifically, the time information of each piece of intermediate caption data is compared with the time information of each corresponding video frame to obtain each corresponding time difference value;
203. and if the time difference value is smaller than the preset difference value, embedding the middle caption data into the video frame.
204. If the time difference is larger than the preset difference, the middle caption data is discarded, empty caption data is constructed, and the empty caption data is embedded into the video frame.
Specifically, the preset difference may be a time constant, for example, the preset difference may be 20ms, if a time difference is less than 20ms, the corresponding intermediate caption data is embedded into the corresponding video frame, and if a time difference is greater than 20ms, the corresponding intermediate caption data is discarded, empty caption data is constructed, and the empty caption data is embedded into the corresponding video frame.
By adjusting the preset difference value, the synchronization between each piece of intermediate caption data and each corresponding video frame can be ensured, the intermediate caption data can be accurately embedded into the video frame, and the intermediate caption data is prevented from being embedded into the video frame which does not correspond.
Referring to fig. 4, fig. 4 shows an implementation flow for acquiring a TS slice and a WebVTT subtitle slice of an HLS stream according to an embodiment of the present application.
As shown in fig. 4, the acquiring of the TS slice and the WebVTT subtitle slice of the HLS stream includes:
301. acquiring a main play list according to the HLS stream;
specifically, the main playlist may be obtained from a data packet or a resource packet based on the HLS stream.
302. Extracting an m3u8 file of the WebVTT subtitle from a main play list;
specifically, the Master Playlist is obtained from the first-level m3u8 file of the main Playlist, and then the m3u8 file of the WebVTT subtitle is obtained from the Master Playlist. The m3u8 file is an index file, the Master Playlist is the file content of the first-level m3u8 file, and the path of the WebVTT subtitle slice file is recorded in the m3u8 file of the WebVTT subtitle.
303. And acquiring the WebVTT subtitle slice according to the m3u8 file of the WebVTT subtitle.
Specifically, a WebVTT subtitle slice file in an m3u8 file of the WebVTT subtitle is downloaded and read to obtain a WebVTT subtitle slice.
The acquisition process of the TS slices of the HLS stream comprises the following steps: the method comprises the steps of obtaining a Master Playlist from a first-level m3u8 file of a main playing list, obtaining a second-level m3u8 file from the Master Playlist, obtaining a Media Playlist from a second-level m3u8 file, downloading a TS slice file in the Media Playlist, and reading to obtain a TS slice. The Media Playlist is the file content of the second level m3u8 file, and the Media Playlist records therein a path of the TS slice file.
In one embodiment, converting the WebVTT subtitle slice to obtain intermediate subtitle data includes: and reading the subtitle content information of the WebVTT subtitle slice to obtain WebVTT subtitle data. And converting the WebVTT subtitle data based on a preset embedded subtitle standard to obtain intermediate subtitle data.
Specifically, the subtitle content information of the WebVTT subtitle slice is read to form WebVTT subtitle data, and the WebVTT subtitle data is placed in a WebVTT subtitle queue for extraction. The embedded caption standard comprises a series of embedded caption standards such as an EIA-708 standard and an EIA-608 standard. And subsequently, extracting WebVTT subtitle data from the WebVTT subtitle queue, and converting the WebVTT subtitle data into subtitle data based on the embedded subtitle standard, namely intermediate subtitle data.
In one embodiment, converting WebVTT subtitle data based on a preset embedded subtitle standard includes: and acquiring WebVTT subtitle control characters and WebVTT subtitle information from the WebVTT subtitle data. And converting the WebVTT subtitle control characters and the WebVTT subtitle information according to a preset embedded subtitle standard to form intermediate subtitle control characters and intermediate subtitle information, and placing the intermediate subtitle information in an intermediate subtitle queue for extraction.
Specifically, the WebVTT caption control character is converted to form an intermediate caption control character based on the caption control character of the preset embedded caption standard, and the WebVTT caption information is converted to form intermediate caption information based on the caption information of the preset embedded caption standard. The WebVTT subtitle control characters and the intermediate subtitle control characters comprise a series of subtitle control characters such as font size, font color and subtitle rolling, and control over WebVTT subtitles and intermediate subtitles is achieved.
In one embodiment, obtaining and outputting a TS stream from a video frame to be embedded includes: and performing code rate output control according to the relation between the local timestamp of the system and the DTS of the embedded video frame to obtain TS stream.
Specifically, the DTS is a decoding timestamp, and this step is to perform rate output control in a CBR manner to obtain a TS stream, so that the video is played more smoothly without a pause phenomenon.
Referring to fig. 5, fig. 5 shows an implementation flow of rate output control according to a relationship between a system local timestamp and a DTS of an embedded video frame, provided in an embodiment of the present application.
As shown in fig. 5, the performing rate output control according to the relation between the system local timestamp and the DTS of the embedded video frame includes:
401. obtaining a phase difference value according to the DTS of the current embedded video frame and a system local time stamp;
specifically, the DTS of the current embedded video frame is converted into a time value of the current embedded video frame, the system local timestamp is converted into a system local time value, then the system local time value is converted into a system local time conversion value in the same unit as the time value of the current embedded video frame, and then a phase difference value is obtained according to the time value of the current embedded video frame and the system local time conversion value.
402. Obtaining a sum value according to the difference value and the DTS of the next frame of the embedded video frame;
specifically, the DTS of the next frame of the embedded video frame is converted into a time value of the next frame of the embedded video frame, and then an added value is obtained according to the difference value and the time value of the next frame of the embedded video frame.
403. If the added value is larger than the local timestamp of the system, outputting the current embedded video frame;
404. and if the added value is smaller than the system local timestamp, waiting for the added value to be equal to the system local timestamp and outputting the current embedded video frame.
Specifically, after the added value is compared with the system local time conversion value, whether the current embedded video frame is output or not is determined, so that code rate output control is performed in a constant code rate mode, a TS (transport stream) is obtained, video playing is smooth, and the phenomenon of blocking is avoided.
In one embodiment, converting the TS slice to obtain the video frame includes: and reading video content information of the TS slices and placing the video content information in a video frame queue. A complete GOP group video frame is selected from the video frame queue and placed in the GOP group queue. The GOP group video frames are extracted from the GOP group queue and sorted according to the PTS of the GOP group video frames.
Specifically, the GOP group is the minimum unit in a video frame, that is, the minimum unit of video playing. The PTS of the GOP group video frame is translated or conveyed by the PTS of the above video frame. Before the ordering, the GOP group video frames placed in the GOP group queue are irregular, and after the ordering is performed according to the PTS of the GOP group video frames, the GOP group video frames are regular, for example, the GOP group video frames can be ordered from small to large according to the numerical value corresponding to the PTS of the GOP group video frames.
Before the intermediate caption data is embedded into the video frame to form the embedded video frame carrying the caption, the intermediate caption data is extracted from the intermediate caption queue and sequenced according to the PTS of the intermediate caption data, for example, the intermediate caption data can be sequenced from small to large according to the numerical value corresponding to the PTS of the intermediate caption data, so as to ensure the synchronization with the GOP group video frame.
Referring to fig. 6, fig. 6 shows a hardware framework of an electronic device provided in an embodiment of the present application.
As shown in fig. 6, the electronic apparatus includes:
a processor 501; and
the memory 502, the memory 502 stores a computer program therein, and the processor 501 is configured to execute the HLS stream-based data processing method by calling the computer program stored in the memory 502.
It is understood that the type of the components of the processor 501 and the memory 502 may be any type according to the requirement, and the present application is not limited thereto.
Wherein, the method comprises the following steps: and acquiring TS slices and WebVTT subtitle slices of the HLS stream. And converting the TS slices to obtain video frames. And converting the WebVTT subtitle slices to obtain intermediate subtitle data. And embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying the caption. And obtaining and outputting the TS stream according to the embedded video frame.
In an embodiment, the processor 501 may be further configured to: after the step of obtaining and outputting the TS stream according to the embedded video frame, the method further includes: and decoding the TS stream to obtain decoded data carrying subtitles.
In an embodiment, the processor 501 may be further configured to: embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying the caption, comprising: time information of the video frame and the middle subtitle data is acquired. And obtaining a time difference value according to the time information of the video frame and the middle caption data. And if the time difference value is smaller than the preset difference value, embedding the middle caption data into the video frame. If the time difference is larger than the preset difference, the middle caption data is discarded, empty caption data is constructed, and the empty caption data is embedded into the video frame.
In an embodiment, the processor 501 may be further configured to: the method for acquiring the TS slices and the WebVTT subtitle slices of the HLS streams comprises the following steps: the main playlist is obtained from the HLS stream. The m3u8 file of the WebVTT subtitle is extracted from the main playlist. And acquiring the WebVTT subtitle slice according to the m3u8 file of the WebVTT subtitle.
In an embodiment, the processor 501 may be further configured to: converting the WebVTT subtitle slice to obtain intermediate subtitle data, which comprises the following steps: and reading the subtitle content information of the WebVTT subtitle slice to obtain WebVTT subtitle data. And converting the WebVTT subtitle data based on a preset embedded subtitle standard to obtain intermediate subtitle data.
In an embodiment, the processor 501 may be further configured to: converting WebVTT subtitle data based on a preset embedded subtitle standard, comprising the following steps: and acquiring WebVTT subtitle control characters and WebVTT subtitle information from the WebVTT subtitle data. And converting the WebVTT subtitle control characters and the WebVTT subtitle information according to a preset embedded subtitle standard to form intermediate subtitle control characters and intermediate subtitle information. And placing the intermediate caption information into an intermediate caption queue for extraction.
In an embodiment, the processor 501 may be further configured to: obtaining and outputting a TS stream according to the embedded video frame, wherein the TS stream comprises: and performing code rate output control according to the relation between the local timestamp of the system and the DTS of the embedded video frame to obtain TS stream.
In an embodiment, the processor 501 may be further configured to: performing code rate output control according to the relation between the system local timestamp and the DTS of the embedded video frame, comprising: and obtaining a phase difference value according to the DTS of the current embedded video frame and the local time stamp of the system. And obtaining an added value according to the difference value and the DTS of the next frame of the embedded video frame. And if the added value is greater than the local timestamp of the system, outputting the current embedded video frame. And if the added value is smaller than the system local timestamp, waiting for the added value to be equal to the system local timestamp and outputting the current embedded video frame.
In an embodiment, the processor 501 may be further configured to: converting the TS slices to obtain video frames, wherein the method comprises the following steps:
and reading video content information of the TS slices and placing the video content information in a video frame queue. A complete GOP group video frame is selected from the video frame queue and placed in the GOP group queue. The GOP group video frames are extracted from the GOP group queue and sorted according to the PTS of the GOP group video frames.
As can be seen from the above, in the electronic device of the present application, the intermediate subtitle data is obtained by converting the WebVTT subtitle slice, the intermediate subtitle data is embedded into the corresponding video frame to form the embedded video frame carrying the subtitle, the embedded video frame is converted to obtain the TS stream, and the TS stream is output to the existing set top box or television device to display the subtitle, so that the conversion display of the WebVTT subtitle in the existing set top box or television device is realized, and the subtitle compatibility of the existing set top box or television device for the video of the HLS stream is improved.
In this embodiment of the application, the electronic device and the HLS-based data processing method in the above embodiments belong to the same concept, and any method step provided in the HLS-based data processing method embodiment may be executed on the electronic device, and a specific implementation process of the method is described in detail in the HLS-based data processing method embodiment, and any combination of the method steps may be adopted to form an optional embodiment of the application, which is not described herein again.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
The embodiments of the present application have been described in detail with reference to the drawings, but the present application is not limited to the above embodiments, and various changes can be made without departing from the spirit of the present application within the knowledge of those skilled in the art.
Claims (10)
1. A method of data processing based on HLS streams, the method comprising:
acquiring TS slices and WebVTT subtitle slices of HLS streams;
converting the TS slices to obtain video frames;
converting the WebVTT subtitle slices to obtain intermediate subtitle data;
embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying captions;
and obtaining and outputting TS stream according to the embedded video frame.
2. The HLS-stream-based data processing method as claimed in claim 1, wherein after the step of obtaining and outputting a TS stream according to the embedded video frame, further comprising:
and decoding the TS stream to obtain decoded data carrying subtitles.
3. The HLS stream-based data processing method of claim 1, wherein the embedding the intermediate caption data into the corresponding video frame to form an embedded video frame carrying captions comprises:
acquiring time information of the video frame and the intermediate caption data;
obtaining a time difference value according to the time information of the video frame and the intermediate caption data;
if the time difference is smaller than a preset difference, embedding the intermediate caption data into the video frame;
and if the time difference is greater than a preset difference, discarding the intermediate caption data, constructing empty caption data, and embedding the empty caption data into the video frame.
4. The data processing method based on the HLS stream as claimed in claim 1, wherein the obtaining of the TS slice and the WebVTT subtitle slice of the HLS stream comprises:
acquiring a main play list according to the HLS stream;
extracting an m3u8 file of the WebVTT subtitle from the main playlist;
and acquiring the WebVTT subtitle slice according to the m3u8 file of the WebVTT subtitle.
5. The HLS-stream-based data processing method of claim 1, wherein said converting the WebVTT subtitle slice to obtain intermediate subtitle data comprises:
reading the subtitle content information of the WebVTT subtitle slice to obtain WebVTT subtitle data;
and converting the WebVTT subtitle data based on a preset embedded subtitle standard to obtain the intermediate subtitle data.
6. The HLS-stream-based data processing method as claimed in claim 5, wherein the converting the WebVTT subtitle data based on the preset embedded subtitle standard comprises:
acquiring WebVTT subtitle control characters and WebVTT subtitle information from the WebVTT subtitle data;
converting the WebVTT subtitle control characters and the WebVTT subtitle information according to a preset embedded subtitle standard to form intermediate subtitle control characters and intermediate subtitle information;
and placing the intermediate caption information into an intermediate caption queue for extraction.
7. The HLS-stream-based data processing method as claimed in claim 1, wherein said obtaining and outputting a TS stream according to the embedded video frame comprises:
and performing code rate output control according to the relation between the system local timestamp and the DTS of the embedded video frame to obtain the TS stream.
8. The HLS-stream-based data processing method of claim 7, wherein said performing rate output control according to the relation of the system local timestamp and the DTS of the embedded video frame comprises:
obtaining a phase difference value according to the DTS of the current embedded video frame and the local timestamp of the system;
obtaining a sum value according to the difference value and the DTS of the next frame of the embedded video frame;
if the added value is larger than the system local timestamp, outputting the current embedded video frame;
and if the added value is smaller than the system local timestamp, waiting for the added value to be equal to the system local timestamp and then outputting the current embedded video frame.
9. The HLS-stream-based data processing method as claimed in claim 1, wherein said converting said TS slices to obtain video frames comprises:
reading video content information of the TS slices and placing the video content information in a video frame queue;
selecting a complete GOP group video frame from the video frame queue and placing the complete GOP group video frame in a GOP group queue;
and extracting GOP group video frames from the GOP group queue and sequencing according to the PTS of the GOP group video frames.
10. An electronic device, characterized in that the electronic device comprises:
a processor; and
memory in which a computer program is stored, the processor being adapted to perform the HLS-stream based data processing method according to any of claims 1-9 by calling the computer program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110765617.2A CN113315931B (en) | 2021-07-06 | 2021-07-06 | HLS stream-based data processing method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110765617.2A CN113315931B (en) | 2021-07-06 | 2021-07-06 | HLS stream-based data processing method and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113315931A CN113315931A (en) | 2021-08-27 |
| CN113315931B true CN113315931B (en) | 2022-03-11 |
Family
ID=77381862
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110765617.2A Active CN113315931B (en) | 2021-07-06 | 2021-07-06 | HLS stream-based data processing method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113315931B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8782722B1 (en) * | 2013-04-05 | 2014-07-15 | Wowza Media Systems, LLC | Decoding of closed captions at a media server |
| CN106331089A (en) * | 2016-08-22 | 2017-01-11 | 乐视控股（北京）有限公司 | Video play control method and system |
| CN106792088A (en) * | 2016-12-15 | 2017-05-31 | 北京奇艺世纪科技有限公司 | The encryption and decryption method and system of a kind of plug-in captions, plug-in caption system |
| CN108055574A (en) * | 2017-11-29 | 2018-05-18 | 上海网达软件股份有限公司 | Media file transcoding generates the method and system of multitone rail multi-subtitle on-demand content |
| CN108174279A (en) * | 2017-12-07 | 2018-06-15 | 四川迪佳通电子有限公司 | Set-top box caption presentation method and system |
| CN109040779A (en) * | 2018-07-16 | 2018-12-18 | 腾讯科技（深圳）有限公司 | Caption content generation method, device, computer equipment and storage medium |
| CN109729420A (en) * | 2017-10-27 | 2019-05-07 | 腾讯科技（深圳）有限公司 | Image processing method and device, mobile terminal and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9888047B2 (en) * | 2014-04-03 | 2018-02-06 | Cisco Technology, Inc. | Efficient on-demand generation of ABR manifests |
-
2021
- 2021-07-06 CN CN202110765617.2A patent/CN113315931B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8782722B1 (en) * | 2013-04-05 | 2014-07-15 | Wowza Media Systems, LLC | Decoding of closed captions at a media server |
| CN106331089A (en) * | 2016-08-22 | 2017-01-11 | 乐视控股（北京）有限公司 | Video play control method and system |
| CN106792088A (en) * | 2016-12-15 | 2017-05-31 | 北京奇艺世纪科技有限公司 | The encryption and decryption method and system of a kind of plug-in captions, plug-in caption system |
| CN109729420A (en) * | 2017-10-27 | 2019-05-07 | 腾讯科技（深圳）有限公司 | Image processing method and device, mobile terminal and computer readable storage medium |
| CN108055574A (en) * | 2017-11-29 | 2018-05-18 | 上海网达软件股份有限公司 | Media file transcoding generates the method and system of multitone rail multi-subtitle on-demand content |
| CN108174279A (en) * | 2017-12-07 | 2018-06-15 | 四川迪佳通电子有限公司 | Set-top box caption presentation method and system |
| CN109040779A (en) * | 2018-07-16 | 2018-12-18 | 腾讯科技（深圳）有限公司 | Caption content generation method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113315931A (en) | 2021-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11557015B2 (en) | System and method of data transfer in-band in video via optically encoded images | |
| WO2017063399A1 (en) | Video playback method and device | |
| KR100998899B1 (en) | Method for service of thumbnail image and broadcasting receiver | |
| KR20130133266A (en) | Systems and methods for adaptive bitrate streaming of media including subtitles | |
| BRPI0706887A2 (en) | "method for suppressing some multimedia content from a multimedia content presentation provided by a remote server to a client multimedia content player and method for applying and synchronizing media content filter data with a multimedia content presentation . " | |
| WO2003017671A2 (en) | Navigation for mpeg streams | |
| US8483053B2 (en) | Information processing device, information processing method, program, and data structure | |
| US20110019870A1 (en) | Embedding interactive data into an audiovisual content by watermarking | |
| JP2006115457A (en) | System and its method for embedding multimedia editing information into multimedia bit stream | |
| US10341631B2 (en) | Controlling modes of sub-title presentation | |
| US20100098161A1 (en) | Video encoding apparatus and video encoding method | |
| US9883216B2 (en) | Method and apparatus for carrying transport stream | |
| KR20110063004A (en) | Apparatus and method for extracting key frames and apparatus and method for recording broadcast signal using thereof | |
| CN114040255A (en) | Live caption generating method, system, equipment and storage medium | |
| JP2005123907A (en) | Data reconstruction apparatus | |
| JP2006270634A (en) | Digital broadcast synchronizing reproducing apparatus, stream synchronization reproducing apparatus, and stream synchronization reproducing system | |
| US8311101B2 (en) | Rate control for real time transcoding of subtitles for application with limited memory | |
| CN113315931B (en) | HLS stream-based data processing method and electronic equipment | |
| US8457468B2 (en) | Method for embedding video annotation data into a coded video stream and video recording device | |
| KR101183861B1 (en) | Method of performing a processing of a multimedia content | |
| US20080137733A1 (en) | Encoding device, decoding device, recording device, audio/video data transmission system | |
| JP2008134825A (en) | Information processor, information processing method and program | |
| JP4764707B2 (en) | Program unit separation device and program unit separation program | |
| JP2007124575A (en) | Recording apparatus, output apparatus, and edit apparatus | |
| JP4114868B2 (en) | Multiplexer and multiplexing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |