CN112954374B - Video data processing method and device, electronic equipment and storage medium - Google Patents
Video data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112954374B CN112954374B CN202110121083.XA CN202110121083A CN112954374B CN 112954374 B CN112954374 B CN 112954374B CN 202110121083 A CN202110121083 A CN 202110121083A CN 112954374 B CN112954374 B CN 112954374B
- Authority
- CN
- China
- Prior art keywords
- additional information
- video
- data packet
- live
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 230000005540 biological transmission Effects 0.000 claims abstract description 87
- 238000012545 processing Methods 0.000 claims abstract description 36
- 230000006835 compression Effects 0.000 claims abstract description 33
- 238000007906 compression Methods 0.000 claims abstract description 33
- 238000000034 method Methods 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 9
- 238000009877 rendering Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000004806 packaging method and process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000012856 packing Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23106—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving caching operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N21/2355—Processing of additional data, e.g. scrambling of additional data or processing content descriptors involving reformatting operations of additional data, e.g. HTML pages
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N21/4355—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream involving reformatting operations of additional data, e.g. HTML pages on a television screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the invention provides a video data processing method, a video data processing device, electronic equipment and a storage medium, and relates to the technical field of Internet. The video data processing method comprises the following steps: sequentially extracting video additional information corresponding to each frame of live image in live video data, and storing the video additional information in a buffer area; obtaining a plurality of frames of video additional information from the buffer area for compression so as to obtain an additional information data packet; and respectively sending the additional information data packet and the live video data to a corresponding client. Therefore, the compression rate of the video additional information is improved, the size of the obtained additional information data packet is reduced, and transmission resources are occupied in the transmission process.
Description
Technical Field
The present invention relates to the field of internet technologies, and in particular, to a video data processing method, a device, an electronic apparatus, and a storage medium.
Background
With the development of internet technology, the playing content of video data is more and more diversified, and particularly in a live broadcast scene, the playing of the video data is not limited to video pictures, and video additional information (such as contour information of a graphic element focused by a user) of the video data is often required to be displayed.
Currently, the video server encodes and packages the extracted video additional information together with the video picture of the video data and transmits the encoded and packaged video additional information to a client used by a user. However, this approach requires coding and packing the resulting video overhead per frame. Obviously, on one hand, the compression workload is increased, and on the other hand, the size of the transmission data packet still has an optimized space, so that unnecessary data transmission resource waste is generated.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a video data processing method, apparatus, electronic device, and storage medium.
In order to achieve the above object, the technical scheme adopted by the embodiment of the invention is as follows:
in a first aspect, the present invention provides a video data processing method, applied to a live broadcast server, where the video data processing method includes: sequentially extracting video additional information corresponding to each frame of live image in live video data, and storing the video additional information in a buffer area; obtaining a plurality of frames of video additional information from the buffer area for compression so as to obtain an additional information data packet; and respectively sending the additional information data packet and the live video data to a corresponding client.
Therefore, in the above embodiment, the video data processing method uses the characteristic that the difference between the video additional information of adjacent multi-frame live images in the same live video data is not large, and selects to pack and compress the video additional information corresponding to the multi-frame live images together, so that compared with the video additional information packed frame by frame, the frequency of packing the video additional information is reduced, and the compression rate is improved. That is, the size of the obtained data packet can be significantly reduced after the data packet is packed and compressed by the scheme in the face of the same number of video additional information. On the basis, the additional information data packet and the live video data obtained after packaging are respectively sent to the client for playing, so that the occupation of transmission resources is reduced while the live video data and the additional information data packet can reach the client.
In an alternative embodiment, the step of obtaining the plurality of frames of the video additional information from the buffer area for compression includes: acquiring transmission delay of the live image transmitted to the client; acquiring the video additional information from the buffer area for compression according to the transmission time delay and the acquisition time stamp of the video additional information; and the acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live image.
It is easy to understand that live video data is collected in real time in a live scene, so that video additional information corresponding to a live image is also extracted in real time. In order to pack the multi-frame real-time extracted video additional information, the real-time extracted video additional information is firstly put into a buffer area so as to be compressed and packed after the multi-frame video additional information is stored in the buffer area. Although the more video additional information compressed together, the higher the compression rate, the more video additional information is obtained at the cost of longer time the video additional information is retained in the buffer. However, the client needs to render the live image in the live video data synchronously with the corresponding video additional information.
Therefore, in the above embodiment, the video additional information is collected, compressed and packaged by utilizing the inherent time delay in the live video data transmission process, so that the video additional information is prevented from being sent to the client later than the corresponding live image, and the live real-time performance is prevented from being affected.
The above-mentioned collection and compression packaging of video additional information by utilizing the inherent delay in the live video data transmission process can be realized by adopting the following two modes:
according to the first mode, calculating the maximum time difference between the video additional information stored in the buffer area according to the acquisition time stamp of the video additional information stored in the buffer area; and under the condition that the maximum time difference is matched with the transmission time delay, taking out the video additional information stored in the buffer area, and performing compression processing.
In the second mode, when the sum of the storage time length of the video additional information written in the buffer area first and the transmission time consumption corresponding to the additional information data packet is equal to the transmission time delay, all the video additional information in the buffer area is taken out and compressed.
In an alternative embodiment, the step of sending the additional information data packet and the live video data to the corresponding client respectively includes: generating a video data packet according to the live image of the live video data; transmitting the video data packet to the client through a video transmission channel; and transmitting the additional information data packet to the client through a data packet transmission channel.
In a second aspect, the present invention provides a video data processing method, applied to a client, the video data processing method including: receiving a video data packet and an additional information data packet corresponding to live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is a data packet generated based on a live image of the live video data; and after the video data packet and the additional information data packet are respectively decoded, acquiring live broadcast images and video additional information with corresponding identifications so as to synchronously render.
In an alternative embodiment, the video data processing method further includes: periodically calculating transmission delay according to the acquired acquisition time stamp and world time of the live image; and feeding back the transmission delay to a live broadcast server to guide the live broadcast server to create the additional information data packet.
In a third aspect, the present invention provides a video data processing apparatus applied to a live server, the video data processing apparatus comprising: the extraction module is used for sequentially extracting video additional information corresponding to each frame of live image in the live video data and storing the video additional information into the buffer area; the compression module is used for acquiring a plurality of frames of the video additional information from the buffer area to compress the video additional information so as to obtain an additional information data packet; and the sending module is used for respectively sending the additional information data packet and the live video data to the corresponding client.
In a fourth aspect, the present invention provides a video data processing apparatus, for application to a client, the video data processing apparatus comprising:
the receiving module is used for receiving the video data packet and the additional information data packet corresponding to the live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is a data packet generated based on a live image of the live video data;
and the rendering module is used for acquiring live broadcast images and video additional information with corresponding identifications after respectively decoding the video data packet and the additional information data packet so as to synchronously render.
In a fifth aspect, the invention provides an electronic device comprising a processor and a memory storing machine executable instructions executable by the processor to implement the method of any of the preceding embodiments.
In a sixth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method according to any of the preceding embodiments.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a live scene schematic diagram provided by an embodiment of the present invention.
Fig. 2 shows a schematic diagram of an electronic device according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating steps of a video data processing method applied to a live server according to an embodiment of the present invention.
Fig. 4 shows an example diagram of encoding video additional information in the related art.
Fig. 5 shows an exemplary diagram of merging and compressing video additional information according to an embodiment of the present invention.
Fig. 6 shows a flow chart of the substeps of step S102 provided by an embodiment of the present invention.
Fig. 7 conceptually illustrates an example diagram of a video data packet and an additional information data packet transmitted to a viewer client.
Fig. 8 is a schematic diagram of a video data processing apparatus applied to a live server according to an embodiment of the present invention.
Fig. 9 is a flowchart illustrating steps of a video data processing method applied to a client according to an embodiment of the present invention.
Fig. 10 is a schematic diagram of a video data processing apparatus applied to a client according to an embodiment of the present invention.
Icon: 100-an electronic device; 110-memory; a 120-processor; 130-a communication module; 400-video data processing means; 401-an extraction module; 402-a compression module; 403-a transmitting module; 500-video data processing means; 501-a receiving module; 502-a rendering module.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention.
It is noted that relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In order to optimize the transmission scheme of the video additional information corresponding to the video data, the transmission scheme of the video additional information corresponding to the live video data in the live scene is particularly adopted. The embodiment of the invention provides a video data processing method, a video data processing device, electronic equipment and a storage medium.
The application scene of the video data processing method and the video data processing device can be a scene in which the image information acquired in real time is required to be obtained. For example, but not limited to, including live scenes, monitored scenes, and the like.
The video data processing method and apparatus, the electronic device 100, and the storage medium provided in the embodiments of the present invention are described below by taking the application of the video data processing method and apparatus to the live broadcast scene shown in fig. 1 as an example. As shown in fig. 1, a live client (also called a host) collects live image frames to generate live video data, and sends the live video data to a video server in the form of a video stream. And after the video server carries out relevant processing on the live video data, the live video data is sent to the corresponding audience client. The viewer client may be a client used by a viewer entering a live room created by a live client.
It should be noted that, the live client and the viewer client are both clients that obtain services from the video server essentially, and differ only in the roles assumed in the live event. Each client can switch between live clients and viewer clients according to the user's selections. For example, user a uses client a for live broadcast, then client a is a live client, and client B used by user B entering the live broadcast room created by user a is a viewer client. After the user A downloads, when the user A enters a live broadcasting room created by the user B, the client B used by the user B which is in live broadcasting is a live broadcasting client, and the client A is a spectator client.
The video data processing method and device provided by the embodiment of the invention can be applied to the live broadcast server and the corresponding audience client in fig. 1.
The live servers may be, but are not limited to, personal computers (personal computer, PCs), servers, distributed deployed computers, and the like. It is to be understood that the server is not limited to a physical server, and may be a virtual machine on a physical server, a virtual machine built on a cloud platform, or a computer that can provide the same function as the server or the virtual machine.
The client may be an intelligent device with image acquisition and display functions, for example, may include a mobile phone, an intelligent home device, a wearable device, an intelligent mobile device, a virtual reality device, an augmented reality device, or the like, or may be a tablet computer, a laptop computer, or a built-in device in a motor vehicle, or the like.
For convenience in describing the hardware environment in which the video data processing method and apparatus described above are operated, both the live broadcast server and the corresponding viewer client will be referred to as the electronic device 100.
Referring to fig. 2, fig. 2 is a block diagram of the electronic device 100. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120, and the communication module 130 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
Wherein the memory 110 is used for storing programs or data. The Memory 110 may be, but is not limited to, a random access Memory 110 (Random Access Memory, RAM), a Read Only Memory 110 (ROM), a programmable Read Only Memory 110 (Programmable Read-Only Memory, PROM), an erasable Read Only Memory 110 (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable Read Only Memory 110 (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
The processor 120 is used to read/write data or programs stored in the memory 110 and perform corresponding functions.
The communication module 130 is configured to establish a communication connection between the electronic device 100 and other communication terminals through the network, and is configured to transmit and receive data through the network.
It should be understood that the structure shown in fig. 2 is merely a schematic diagram of the structure of the electronic device 100, and that the electronic device 100 may further include more or fewer components than those shown in fig. 2, or have a different configuration than that shown in fig. 2. The components shown in fig. 2 may be implemented in hardware, software, or a combination thereof.
First embodiment
Referring to fig. 3, fig. 3 illustrates a video data processing method according to an embodiment of the present invention. The video data processing method may be applied to a live server, as shown in fig. 3, and may include the following steps:
step S101, extracting video additional information corresponding to each frame of live image in the live video data in sequence, and storing the video additional information in a buffer area.
The live video data may be a video stream received by a live server. In a live broadcast scene, the live broadcast video data may be a video stream collected and sent by a live broadcast client. In the monitoring scene, the live video data may be a video stream collected by a camera installed in the monitored site.
The live video data consists of a plurality of live images which are sequentially sequenced according to the acquisition time.
The video additional information may be information that is extracted based on the live image and is of interest to the user, or information that is related to the live image may be written. For example, in a live broadcast with a goods scene, the video additional information may be clothes appearing in the live broadcast image, a purchasing link of the appearing clothes to be added to the live broadcast image, and the like. Of course, the description information for the live image may also be, for example, a residual error between the live image and a last live image, and texture detail area information in a video frame.
The buffer may be a first-in first-out storage area for temporary data storage.
In some embodiments, the corresponding video additional information may be extracted based on each frame of live image of the live video data in turn. Every time video additional information corresponding to one frame of live image (hereinafter referred to as one frame of video additional information for convenience of description) is extracted, the video additional information is written into the buffer area.
Unlike the related art, the video additional information is not directly encoded and packed together with the live image after being extracted, but is written into the buffer. It can be understood that the live video data has a certain real-time property, each frame of live image in the live video data is collected in real time, and corresponding to the live image, each frame of video additional information is extracted in real time, so that in order to accumulate multi-frame video additional information, the extracted video additional information needs to be written into a buffer area.
Step S102, multi-frame video additional information is obtained from the buffer area for compression to obtain additional information data packets.
The additional information data packet is a data packet obtained by compressing the additional information according to the video.
In some embodiments, after the multi-frame video additional information is buffered in the buffer, the buffered multi-frame video additional information may be fetched and combined, compressed, and packaged to obtain an additional information packet.
In the related art, since the video additional information needs to be encoded and packaged together with the live image, it is equivalent to generating an additional information data packet corresponding to each frame of video additional information extracted, as shown in fig. 4. Although the transmission resources required for a single additional information data packet are not large, the compression rate of the single additional information data is not high, so that the transmission resources occupied by a plurality of additional information data are large.
Since the picture contents of adjacent live images are not different, the corresponding video additional information extracted from the adjacent live images is not different. The multiple frames of video additional information are combined and compressed together to obtain an overall additional information data packet, as shown in fig. 5. Obviously, compared with the plurality of additional information data in fig. 4, the overall additional information data packet compression rate is improved, the corresponding data volume is smaller, and the required transmission resources are less.
In one embodiment, the step S102 may be that the number of the video additional information stored in the buffer exceeds a specified threshold, and then all the video additional information in the buffer is fetched, combined, compressed, and packed to obtain the additional information data packet.
Although the more video additional information is compressed together, the higher the compression rate is. However, collecting more video additional information results in longer retention of the video additional information in the buffer. However, live images in live video data and corresponding video overhead information often also require synchronized rendering. Although a certain transmission delay exists in the process of transmitting live video data to the live server from the live server, forwarding the live video data to the audience client and playing the live video data, if the video additional information is transmitted to the client later than the corresponding live image, the delay of live broadcasting is further increased.
Therefore, in the process of collecting the video additional information by using the buffer, the propagation delay of the live video data needs to be considered. I.e. the propagation delay is used to trigger the packing of the additional information for the video in the buffer.
That is, as another embodiment, as shown in fig. 6, the step S102 may further include the following sub-steps:
and step S102-1, acquiring the transmission delay of the live image transmitted to the client.
In some embodiments, the above sub-step S102-1 may determine the corresponding transmission delay based on the data transmission speed measured by the live server and the viewer client.
In other embodiments, the step S102-1 may further reversely calculate the corresponding transmission delay according to the time point after each frame of live image frame fed back by the viewer client is decoded and the acquisition time point of the live image.
And step S102-2, according to the transmission time delay and in combination with the acquisition time stamp of the video additional information, acquiring the video additional information from the buffer area for compression.
The acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live image.
As one way, the above sub-step S102-2 may be:
(1) And calculating the maximum time difference between the video additional information stored in the buffer area according to the acquisition time stamp of the video additional information stored in the buffer area. It can be understood that the buffer area stores the video additional information in a first-in first-out mode, so that the acquisition time difference between the video additional information which is firstly entered and the video additional information which is finally entered in the buffer area is calculated, and the maximum time difference can be obtained.
It should be noted that the processing time consumption from the extraction of the video additional information to the storage of the video additional information in the buffer area is almost the same for each frame of live image, that is, the corresponding acquisition time difference between the acquisition time stamps of the two video additional information written in the buffer area is also represented as the time difference of the two video additional information written in the buffer area. Thus, in case of continuous processing of the video stream, the corresponding maximum time difference in the buffer may characterize the residence time of the first video additional information in the buffer. Of course, the maximum time difference increases with the newly written video additional information in the buffer.
(2) And under the condition that the maximum time difference is matched with the transmission delay, taking out the video additional information stored in the buffer memory area, and carrying out compression processing.
In some embodiments, the maximum time difference and the transmission delay match may be that a difference between the maximum time difference and the transmission delay does not exceed a maximum transmission time consumption of transmitting the additional information data packet.
As another way, the above sub-step S102-2 may be: and under the condition that the sum of the storage time length of the video additional information written in the buffer memory area at first and the transmission time consumption corresponding to the additional information data packet is equal to the transmission time delay, taking out all the video additional information in the buffer memory area, and carrying out merging, compressing and packaging treatment.
It will be appreciated that after the video additional information in the buffer is retrieved, the buffer is also emptied. And then, continuously receiving and caching new video additional information.
In addition, when the maximum time difference is not matched with the transmission delay, but the video additional information corresponding to the last frame of live image is received, the video additional information stored in the buffer area can be triggered to be taken out, and compression processing is performed.
Step S103, the additional information data packet and the live video data are respectively sent to the corresponding client.
In some embodiments, the additional information data packets and live video data may be transmitted to the corresponding viewer client using different transmission links.
Alternatively, the video data packet may be generated from live images of live video data. As shown in fig. 7, the video data packets are transmitted to the corresponding viewer client through the video transmission channel. After generating the additional information data packet, the additional information data packet is transmitted to the corresponding audience client through the data packet transmission channel. There is no necessary sequence between the two transmissions.
In order to perform the corresponding steps in the above embodiments and the various possible ways, an implementation of the video data processing apparatus 400 is given below, and alternatively, the video data processing apparatus 400 may employ the device structure of the electronic device 100 shown in fig. 2 and described above. Further, referring to fig. 8, fig. 8 is a functional block diagram of a video data processing apparatus 400 according to an embodiment of the present invention. It should be noted that, the basic principle and the technical effects of the video data processing apparatus 400 provided in this embodiment are the same as those of the foregoing embodiments, and for brevity, reference may be made to the corresponding content in the foregoing embodiments. The video data processing apparatus 400 includes: an extraction module 401, a compression module 402 and a transmission module 403.
The extracting module 401 is configured to sequentially extract video additional information corresponding to each frame of live image in the live video data, and store the video additional information in the buffer.
In some embodiments, the step S101 may be performed by the extraction module 401.
And the compression module 402 is configured to obtain multiple frames of the video additional information from the buffer area, and compress the video additional information to obtain an additional information data packet.
In some embodiments, step S102 described above may be performed by the compression module 402.
And the sending module 403 is configured to send the additional information data packet and the live video data to corresponding clients respectively.
In some embodiments, step S103 described above may be performed by the transmitting module 403.
Second embodiment
Referring to fig. 9, fig. 9 shows a video data processing method according to an embodiment of the present invention. The video data processing method may be applied to a client, as shown in fig. 9, and may include the steps of:
step S201, receiving a video data packet and an additional information data packet corresponding to live video data.
The additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from live video data.
The video data packet is a data packet generated based on a live image of the live video data.
In some embodiments, the generation process of the additional information data packet described above may refer to the description in the first embodiment.
Step S202, after the video data packet and the additional information data packet are decoded respectively, a live image with a corresponding identifier and video additional information are obtained for synchronous rendering.
The corresponding identifier is used for identifying live broadcast images and video additional information with corresponding relations. There is generally a correspondence between the video additional information and the live image from which the video additional information is extracted.
In some embodiments, the corresponding identifier may be the same acquisition timestamp, that is, the live image and the video additional information have the same acquisition timestamp, and the representation has the corresponding identifier therebetween.
In other embodiments, the corresponding identifier may also be a live image frame number written in the video additional information, so that it is convenient to determine live images and video additional information with a corresponding relationship.
In some embodiments, the video data packets are decoded upon receipt thereof. Similarly, the additional information data packet is decoded immediately after the additional information data packet is received. And determining the live image and the video additional information with the corresponding identifications in the decoded live image and video additional information so as to synchronously render.
In some embodiments, the video data processing method may further include:
step S301, periodically calculating transmission delay according to the acquired time stamp and world time of the obtained live image.
The world time may be the objective time for the viewing client to decompress out the live image. And calculating the transmission delay through the acquisition time stamp of the live image and the corresponding universal time record.
In other embodiments, in order to reduce the transmission pressure of the live video data, the acquisition time stamp corresponding to the live image may be extracted as the corresponding video additional information. For example, the video additional information includes a frame number and an acquisition timestamp of the corresponding live image. Therefore, the acquisition time stamp can be periodically acquired from the video additional information, the world time of the decoded live image corresponding to the frame number of the video additional information is queried, and finally, the transmission time delay is calculated according to the acquisition time stamp and the corresponding world time.
Step S302, feeding back the transmission delay to the live broadcast server to guide the live broadcast server to create the additional information data packet.
It should be noted that, the step S102 in the first embodiment may be referred to for guiding the live server to create the additional information data packet by using the transmission delay, which is not described herein.
In order to perform the corresponding steps in the above embodiments and the various possible ways, an implementation of the video data processing apparatus 500 is given below, and alternatively, the video data processing apparatus 500 may employ the device structure of the electronic device 100 shown in fig. 2 and described above. Further, referring to fig. 10, fig. 10 is a functional block diagram of a video data processing apparatus 500 according to an embodiment of the present invention. It should be noted that, the basic principle and the technical effects of the video data processing apparatus 500 provided in this embodiment are the same as those of the foregoing embodiments, and for brevity, reference may be made to the corresponding content in the foregoing embodiments. The video data processing apparatus 500 includes: the receiving module 501 and the rendering module 502.
The receiving module 501 is configured to receive a video data packet and an additional information data packet corresponding to live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is a data packet generated based on a live image of the live video data.
In some embodiments, the step S201 may be performed by the receiving module 501.
And the rendering module 502 is configured to obtain a live image and video additional information with corresponding identifiers after decoding the video data packet and the additional information data packet respectively, so as to synchronously render.
In some embodiments, step S202 described above may be performed by rendering module 502.
In some embodiments, the video data processing apparatus further comprises:
and the calculation module is used for periodically calculating the transmission time delay according to the acquired acquisition time stamp of the live image and the world time of the decoded live image.
And the feedback module is used for feeding back the transmission delay to the live broadcast server so as to guide the live broadcast server to create the additional information data packet.
Alternatively, the above modules may be stored in the memory 110 shown in fig. 2 or solidified in an Operating System (OS) of the electronic device 100 in the form of software or Firmware (Firmware), and may be executed by the processor 120 in fig. 2. Meanwhile, data, codes of programs, and the like, which are required to execute the above-described modules, may be stored in the memory 110.
In summary, the embodiment of the invention provides a video data processing method, a device, an electronic device and a storage medium. The video data processing method comprises the steps of sequentially extracting video additional information corresponding to each frame of live image in live video data and storing the video additional information into a buffer area; obtaining a plurality of frames of video additional information from the buffer area for compression so as to obtain an additional information data packet; and respectively sending the additional information data packet and the live video data to a corresponding client. And the additional information data packet and the live video data obtained after the packaging are respectively sent to the client for playing, so that the occupation of transmission resources is reduced while the live video data and the additional information data packet can reach the client.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present invention may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A video data processing method, applied to a live server, comprising:
sequentially extracting video additional information corresponding to each frame of live image in live video data, and storing the video additional information in a buffer area;
acquiring the transmission time delay of the live image transmitted to a client;
acquiring the video additional information from the buffer area according to the transmission time delay and combining with the acquisition time stamp of the video additional information to compress the video additional information so as to obtain an additional information data packet; the acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live image;
generating a video data packet according to the live image of the live video data;
the video data packet is sent to the client through a video transmission channel, and the additional information data packet is transmitted to the client through a data packet transmission channel;
the step of acquiring the video additional information from the buffer area for compression according to the transmission delay and in combination with the acquisition time stamp of the video additional information comprises the following steps:
calculating the maximum time difference between the video additional information stored in the buffer area according to the acquisition time stamp of the video additional information stored in the buffer area; under the condition that the maximum time difference is matched with the transmission time delay, taking out the video additional information stored in the buffer area, and performing compression processing; wherein time is consumed when a difference between the transmission delay and the maximum time difference exceeds a maximum transmission of the additional information data packet, the maximum time difference being matched to the transmission delay;
or, the step of acquiring the video additional information from the buffer area for compression according to the transmission delay and in combination with the acquisition time stamp of the video additional information includes:
and under the condition that the sum of the storage time length of the video additional information written in the buffer memory area at first and the transmission time consumption corresponding to the additional information data packet is equal to the transmission time delay, taking out all the video additional information in the buffer memory area, and performing compression processing.
2. A video data processing method, applied to a client, comprising:
receiving a video data packet and an additional information data packet corresponding to live video data sent by a live server; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is a data packet generated based on a live image of the live video data;
after the video data packet and the additional information data packet are respectively decoded, acquiring live broadcast images and video additional information with corresponding identifications so as to synchronously render;
periodically calculating transmission delay according to the acquired acquisition time stamp of the live image and the world time of the decoded live image;
feeding back the transmission delay to the live broadcast server to guide the live broadcast server to create the additional information data packet;
the live broadcast server sequentially extracts video additional information corresponding to multi-frame live broadcast images and stores the video additional information into a buffer area, and the live broadcast server creates the additional information data packet in the following mode:
calculating the maximum time difference between the video additional information stored in the buffer area according to the acquisition time stamp of the video additional information stored in the buffer area; under the condition that the maximum time difference is matched with the transmission time delay, taking out the video additional information stored in the buffer area, and performing compression processing to obtain the additional information data packet; wherein time is consumed when a difference between the transmission delay and the maximum time difference exceeds a maximum transmission of the additional information data packet, the maximum time difference being matched to the transmission delay;
or alternatively, the process may be performed,
and under the condition that the sum of the storage time length of the video additional information written in the buffer memory area at first and the transmission time consumption corresponding to the additional information data packet is equal to the transmission time delay, taking out all the video additional information in the buffer memory area, and compressing to obtain the additional information data packet.
3. A video data processing apparatus, applied to a live server, comprising:
the extraction module is used for sequentially extracting video additional information corresponding to each frame of live image in the live video data and storing the video additional information into the buffer area;
a compression module for:
acquiring the transmission time delay of the live image transmitted to a client; acquiring the video additional information from the buffer area according to the transmission time delay and combining with the acquisition time stamp of the video additional information to compress the video additional information so as to obtain an additional information data packet; the acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live image;
a sending module, configured to: generating a video data packet according to the live image of the live video data; the video data packet is sent to the client through a video transmission channel, and the additional information data packet is transmitted to the client through a data packet transmission channel;
the compression module is specifically configured to:
calculating the maximum time difference between the video additional information stored in the buffer area according to the acquisition time stamp of the video additional information stored in the buffer area; under the condition that the maximum time difference is matched with the transmission time delay, taking out the video additional information stored in the buffer area, and performing compression processing; wherein time is consumed when a difference between the transmission delay and the maximum time difference exceeds a maximum transmission of the additional information data packet, the maximum time difference being matched to the transmission delay;
or alternatively, the process may be performed,
and under the condition that the sum of the storage time length of the video additional information written in the buffer memory area at first and the transmission time consumption corresponding to the additional information data packet is equal to the transmission time delay, taking out all the video additional information in the buffer memory area, and performing compression processing.
4. A video data processing apparatus, characterized by being applied to a client, comprising:
the receiving module is used for receiving the video data packet and the additional information data packet corresponding to the live video data sent by the live server; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is a data packet generated based on a live image of the live video data;
the rendering module is used for acquiring live broadcast images and video additional information with corresponding identifications after respectively decoding the video data packet and the additional information data packet so as to synchronously render the live broadcast images and the video additional information;
the calculation module is used for periodically calculating the transmission time delay according to the acquired acquisition time stamp of the live image and the world time of the decoded live image;
the feedback module is used for feeding back the transmission delay to the live broadcast server so as to guide the live broadcast server to create the additional information data packet;
the live broadcast server sequentially extracts video additional information corresponding to multi-frame live broadcast images and stores the video additional information into a buffer area, and the live broadcast server creates the additional information data packet in the following mode:
calculating the maximum time difference between the video additional information stored in the buffer area according to the acquisition time stamp of the video additional information stored in the buffer area; under the condition that the maximum time difference is matched with the transmission time delay, taking out the video additional information stored in the buffer area, and performing compression processing to obtain the additional information data packet; wherein time is consumed when a difference between the transmission delay and the maximum time difference exceeds a maximum transmission of the additional information data packet, the maximum time difference being matched to the transmission delay;
or alternatively, the process may be performed,
and under the condition that the sum of the storage time length of the video additional information written in the buffer memory area at first and the transmission time consumption corresponding to the additional information data packet is equal to the transmission time delay, taking out all the video additional information in the buffer memory area, and compressing to obtain the additional information data packet.
5. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the method of any one of claims 1 or 2.
6. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any one of claims 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110121083.XA CN112954374B (en) | 2021-01-28 | 2021-01-28 | Video data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110121083.XA CN112954374B (en) | 2021-01-28 | 2021-01-28 | Video data processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112954374A CN112954374A (en) | 2021-06-11 |
| CN112954374B true CN112954374B (en) | 2023-05-23 |
Family
ID=76238981
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110121083.XA Active CN112954374B (en) | 2021-01-28 | 2021-01-28 | Video data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112954374B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114173171A (en) * | 2021-12-08 | 2022-03-11 | 上海智驾汽车科技有限公司 | Multi-channel big data information synchronous storage and playback method |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017206398A1 (en) * | 2016-05-31 | 2017-12-07 | 乐视控股（北京）有限公司 | Method and device for video sharing |
| CN107809682A (en) * | 2017-11-14 | 2018-03-16 | 北京威速科技有限公司 | It is a kind of to adjust the video flow control method for sending amount of buffered data |
| CN108668158A (en) * | 2017-03-31 | 2018-10-16 | 武汉斗鱼网络科技有限公司 | A kind of barrage synchronous method, client and barrage server |
| CN108989907A (en) * | 2018-08-22 | 2018-12-11 | 深圳市网心科技有限公司 | Live data streams processing method and system, electronic device and computer readable storage medium |
| CN110392274A (en) * | 2019-07-17 | 2019-10-29 | 咪咕视讯科技有限公司 | A kind of information processing method, equipment, client, system and storage medium |
| CN111010614A (en) * | 2019-12-26 | 2020-04-14 | 北京奇艺世纪科技有限公司 | Method, device, server and medium for displaying live caption |
| CN111083515A (en) * | 2019-12-31 | 2020-04-28 | 广州华多网络科技有限公司 | Method, device and system for processing live broadcast content |
| WO2020103203A1 (en) * | 2018-11-19 | 2020-05-28 | 网宿科技股份有限公司 | Method and apparatus for synchronizing whiteboard with video, computing device, and storage medium |
| US10715860B1 (en) * | 2017-11-29 | 2020-07-14 | Twitch Interactive, Inc. | Video stream with additional content areas |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101442836B1 (en) * | 2008-01-07 | 2014-11-04 | 삼성전자주식회사 | Method for providing additional information of video using visible communication and apparatus for the same |
| CN103905846B (en) * | 2012-12-25 | 2017-12-08 | 中国电信股份有限公司 | Content delivery method and server based on IPTV |
| CN105898362A (en) * | 2015-11-25 | 2016-08-24 | 乐视网信息技术（北京）股份有限公司 | Video content retrieval method and device |
| CN112272303A (en) * | 2020-09-23 | 2021-01-26 | 北京五八信息技术有限公司 | Live content display method and device, electronic equipment and storage medium |
-
2021
- 2021-01-28 CN CN202110121083.XA patent/CN112954374B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017206398A1 (en) * | 2016-05-31 | 2017-12-07 | 乐视控股（北京）有限公司 | Method and device for video sharing |
| CN108668158A (en) * | 2017-03-31 | 2018-10-16 | 武汉斗鱼网络科技有限公司 | A kind of barrage synchronous method, client and barrage server |
| CN107809682A (en) * | 2017-11-14 | 2018-03-16 | 北京威速科技有限公司 | It is a kind of to adjust the video flow control method for sending amount of buffered data |
| US10715860B1 (en) * | 2017-11-29 | 2020-07-14 | Twitch Interactive, Inc. | Video stream with additional content areas |
| CN108989907A (en) * | 2018-08-22 | 2018-12-11 | 深圳市网心科技有限公司 | Live data streams processing method and system, electronic device and computer readable storage medium |
| WO2020103203A1 (en) * | 2018-11-19 | 2020-05-28 | 网宿科技股份有限公司 | Method and apparatus for synchronizing whiteboard with video, computing device, and storage medium |
| CN110392274A (en) * | 2019-07-17 | 2019-10-29 | 咪咕视讯科技有限公司 | A kind of information processing method, equipment, client, system and storage medium |
| CN111010614A (en) * | 2019-12-26 | 2020-04-14 | 北京奇艺世纪科技有限公司 | Method, device, server and medium for displaying live caption |
| CN111083515A (en) * | 2019-12-31 | 2020-04-28 | 广州华多网络科技有限公司 | Method, device and system for processing live broadcast content |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112954374A (en) | 2021-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106162235B (en) | For the method and apparatus of Switch Video stream | |
| CN110868600B (en) | Target tracking video plug-flow method, display method, device and storage medium | |
| US11729465B2 (en) | System and method providing object-oriented zoom in multimedia messaging | |
| EP2940940B1 (en) | Methods for sending and receiving video short message, apparatus and handheld electronic device thereof | |
| CN111866525A (en) | Multi-view video playing control method and device, electronic equipment and storage medium | |
| CN109040786B (en) | Camera data transmission method, device and system and storage medium | |
| CN112019905A (en) | Live broadcast playback method, computer equipment and readable storage medium | |
| CN111093094A (en) | Video transcoding method, device and system, electronic equipment and readable storage medium | |
| CN113542795A (en) | Video processing method and device, electronic equipment and computer readable storage medium | |
| CN110784757A (en) | Game live broadcast method and device in live broadcast client, medium and electronic equipment | |
| CN112423140A (en) | Video playing method and device, electronic equipment and storage medium | |
| CN112954374B (en) | Video data processing method and device, electronic equipment and storage medium | |
| CN105122818A (en) | Computer system, distribution control system, distribution control method, and computer-readable storage medium | |
| JP2007336263A (en) | Image processing method, apparatus, and program | |
| TWI806479B (en) | Method of coding and decoding point cloud data, equipment, computer readable storage medium, and electronic device | |
| CN110139128B (en) | Information processing method, interceptor, electronic equipment and storage medium | |
| CN112153401B (en) | Video processing method, communication device and readable storage medium | |
| CN113973224A (en) | Method for transmitting media information, computing device and storage medium | |
| CN112235600B (en) | Method, device and system for processing video data and video service request | |
| CN111093091A (en) | Video processing method, server and system | |
| CN111314350A (en) | Image storage system, storage method, calling system and calling method | |
| CN113660540B (en) | Image information processing method, system, display method, device and storage medium | |
| CN115243053B (en) | Point cloud encoding and decoding method and related equipment | |
| CN109417585B (en) | Method, system and computer readable storage medium for image transmission, image compression and image restoration | |
| CN111800649A (en) | Method and device for storing video and method and device for generating video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |