CN116711298A - Data stream transmission method and device - Google Patents
Data stream transmission method and device Download PDFInfo
- Publication number
- CN116711298A CN116711298A CN202180088320.5A CN202180088320A CN116711298A CN 116711298 A CN116711298 A CN 116711298A CN 202180088320 A CN202180088320 A CN 202180088320A CN 116711298 A CN116711298 A CN 116711298A
- Authority
- CN
- China
- Prior art keywords
- video
- stream
- transmitted
- data stream
- application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 249
- 238000000034 method Methods 0.000 title claims abstract description 119
- 238000012545 processing Methods 0.000 claims description 29
- 238000001514 detection method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 10
- 230000015654 memory Effects 0.000 description 23
- 230000006870 function Effects 0.000 description 22
- 238000004891 communication Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000013461 design Methods 0.000 description 7
- 238000005192 partition Methods 0.000 description 5
- 210000000988 bone and bone Anatomy 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000000926 separation method Methods 0.000 description 4
- 229920001621 AMOLED Polymers 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 235000019800 disodium phosphate Nutrition 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 239000002096 quantum dot Substances 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
- H04N5/08—Separation of synchronising signals from picture signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/08—Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band, e.g. by time division
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The application discloses a data stream transmission method and a data stream transmission device, relates to the field of video stream transmission, and solves the problem of higher transmission delay of video streams in the prior art. The method is executed by a video acquisition device, the video acquisition device is connected with a video display device through a first circuit and a second circuit respectively, the first circuit is used for transmitting control instructions of the video acquisition device and the video display device, and the method comprises the following steps: the video acquisition device generates a plurality of application video streams according to the original video streams acquired by the video acquisition device, and splices the application video streams to obtain a data stream to be transmitted, and then the video acquisition device sends the data stream to be transmitted to the video display device through a second line. The transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line.
Description
The present application relates to the field of data transmission, and in particular, to a data stream transmission method and apparatus.
With the development of intelligent services, televisions can provide users with various types of services such as multi-party video call, human body perception, gesture manipulation, exercise, remote home monitoring, etc. on the television by using artificial intelligence (artificial intelligence, AI) technology. In general, in the above-mentioned service, a camera transmits a control instruction and an original video stream collected by the camera to a television through a universal serial bus (universal serial bus, USB), the television converts the original video stream according to the control instruction to obtain a plurality of application video streams with different resolutions, and further, the television displays the plurality of application video streams, and a user can view a plurality of video information obtained according to the original video streams through the television.
The conversion of the original video stream occupies most of computing resources of the television main chip, and the playing of the application video stream is easy to be blocked due to the lower performance of the television main chip, so that the video watching experience of a user is reduced. In the current technical scheme, in order to avoid the blocking of the playing of the video stream, the camera converts the original video stream acquired by the camera to obtain a plurality of application video streams, and the camera transmits the application video streams to the television through the USB, so that the calculation resources occupied by the conversion of the original video stream by the main chip of the television are reduced. However, the transmission capability of USB is small, resulting in USB having a high latency when transmitting multiple video streams.
Therefore, how to reduce the transmission delay of the video stream is a problem to be solved.
Disclosure of Invention
The application provides a data stream transmission method and a data stream transmission device, which solve the problem of higher transmission delay of a video stream.
In order to achieve the above purpose, the present application adopts the following technical scheme.
In a first aspect, the present application provides a data streaming method performed by a video acquisition device or the method may be applied to a terminal device that may support implementation of the method, e.g. the terminal device comprises a system-on-chip. In one possible design, the video capturing apparatus is connected to the video display apparatus through a first line and a second line, respectively, the first line being used for transmitting control instructions of the video capturing apparatus and the video display apparatus, the method comprising: the video acquisition device generates a plurality of application video streams according to the original video streams acquired by the video acquisition device, and splices the application video streams to obtain a data stream to be transmitted, and then the video acquisition device sends the data stream to be transmitted to the video display device through a second line. The transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line.
Therefore, the video acquisition device transmits the control instruction through the first line, and the data stream to be transmitted is sent to the video display device through the second line, so that the separation transmission of the control instruction and the data stream is realized, and the problem that the transmission delay of the data stream is higher because the control instruction occupies the bandwidth of the line for transmitting the data stream when the control instruction and the data stream are transmitted through the same line is avoided. According to the data stream transmission method provided by the embodiment of the application, the data stream is transmitted by the exclusive second line, so that the transmission delay for transmitting the data stream to be transmitted is reduced, and the transmission rate for transmitting the data stream to be transmitted is improved; in addition, in the process that the video acquisition device sends the data stream to be transmitted to the video display device through the second line, the control instruction sent by the video display device can be received through the first line, and the video display device does not need to send the control instruction to the video acquisition device after the transmission process of the data stream to be transmitted is finished, so that the transmission time delay of the control instruction is effectively reduced. In addition, compared with the prior art that only video streams can be transmitted independently (after one video stream is transmitted, the other video stream is transmitted), in the data stream transmission method used in the embodiment of the application, the video acquisition device can splice a plurality of application video streams to obtain the data stream to be transmitted, and the data stream to be transmitted is put into the second line for transmission, so that the video acquisition device does not need to transmit each application video stream independently, and the transmission delay of the video streams is reduced. In addition, since the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line, the video acquisition device can transmit the data stream to be transmitted according to the maximum transmission bandwidth of the second line, thereby avoiding transmission delay caused by insufficient transmission bandwidth of the second line.
Wherein, the maximum transmission bandwidth of the second line indicates the maximum data amount of the video stream that the second line can transmit in a unit time. Here, taking an example in which N frames of images with the same resolution are required to be transmitted in a unit time, the maximum data amount=resolution of a single frame image×n, N is a positive integer, and the resolution of the image may be expressed in units of pixels, for example, the resolution of the image is 1080P (1920×1080).
The transmission bandwidth required by the data stream to be transmitted indicates the data amount required to be transmitted by all application video streams and description information in the data stream to be transmitted in unit time, and the description information can be used for describing resolution information, exposure information, white balance information, focusing information and the like of the application video streams.
In one possible example, the plurality of application video streams included in the data stream to be transmitted may be different kinds of video streams. For example, the plurality of application video streams includes at least one video stream of a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitor stream, a time-of-flight ranging stream, or a gesture control stream.
The user can view different video information of the original video stream by using the video display device, for example, the user can view the preview stream and the photographing stream in the video call, and view face recognition information (for example, classifying the face of the user) obtained according to the original video stream in the video call process, so that the user experience of viewing the video is improved.
In an alternative implementation, the first line is a USB and the second line is a high definition multimedia interface (high definition multimedia interface, HDMI) line.
Compared with the prior art, the video stream and the control command are transmitted by multiplexing the same transmission line (USB) between the camera and the television, wherein the control command needs to be transmitted in a two-way, and the transmission of the video stream is unidirectional, and in the process that the camera transmits the video stream to the television through the USB, the television cannot transmit the control command to the camera through the USB, so that the control command opposite to the transmission direction of the video stream has higher transmission delay. By using the data stream transmission method provided by the embodiment of the application, the video acquisition device transmits the control instruction through the USB, the data stream to be transmitted is transmitted through the HDMI, the separation transmission of the data stream and the control instruction is realized, the video display device does not need to send the control instruction to the video acquisition device after the transmission process of the data stream to be transmitted is finished, that is, in the process that the video acquisition device sends the data stream to be transmitted to the video display device through the HDMI, the video display device can send the control instruction to the video acquisition device through the USB, and the transmission time delay of the control instruction is reduced. In addition, the transmission bandwidth of the USB in the prior art can only reach 400MB/s (megabytes/s), but the HDMI line used for transmitting the data stream in the present application has a larger data stream transmission capability, such as 10.2 gigabits/s (giga bits per second, gpbs) =1275 MB/s, so the data stream transmission method provided in the embodiment of the present application uses HDMI to replace the USB in the prior art, which can improve the transmission speed of the data stream and reduce the transmission delay of the data stream.
In another alternative implementation, if the second line is an HDMI line, the format of the application video stream may be YUV format. Wherein the Y information indicates brightness (luminence) of each frame image in the application video stream, and the U and V information indicates color information (chromance) of each frame image in the application video stream. Since the video stream in YUV format is a lossless video stream, the transmission bandwidth of USB in the prior art is smaller, resulting in higher transmission delay in YUV format. According to the data stream transmission method provided by the embodiment of the application, the second line is the HDMI line, and the transmission bandwidth of the HDMI line is larger than that of the USB, so that the transmission time delay of the video acquisition device for sending the data stream to be transmitted to the video display device can be reduced.
In addition, in the prior art, in the process of transmitting a plurality of application video streams by using a USB, the camera encodes the application video streams, so as to compress the application video streams to obtain a data stream to be transmitted, wherein the data stream to be transmitted comprises encoding information, and the encoding information indicates the encoding mode of the camera on the application video streams; because the video stream in the data stream to be transmitted is compressed, the video stream included in the data stream to be transmitted has a loss compared with the application video stream, and further, the video stream played on the television is not completely consistent with the application video stream, thereby reducing the playing effect of the video stream. In the data stream transmission method provided by the embodiment of the application, under the condition that the second line is an HDMI line and the format of the application video stream is a YUV format, the video acquisition device does not need to encode the application video stream, and no encoding information is generated, so that fields for describing the format of the application video stream in the data stream to be transmitted are reduced, redundancy of the data stream to be transmitted is reduced, the data amount of the data stream to be transmitted is reduced, the video acquisition device sends the data stream to be transmitted to the video display device by using HDMI, and further, the speed for transmitting the data stream to be transmitted is improved, and the transmission time delay of the data stream to be transmitted is reduced. In addition, the video acquisition device can transmit the data stream in YUV format by using the exclusive second line, so that no difference exists between the video stream played by the video display device and the application video stream generated by the video acquisition device, the effect of playing the video stream by the video display device is improved, and the experience of watching the video by a user is further improved.
In another optional implementation manner, the video acquisition device may further receive an acquisition instruction sent by the video display device through the first line, and acquire an original video stream according to the acquisition instruction; the video acquisition device also processes the original video stream according to the acquisition instruction to obtain a plurality of application video streams. For example, the video capturing device may generate the plurality of application video streams based on a plurality of capturing instructions, for example, the video display device generates a plurality of capturing instructions according to gesture information of a user, the plurality of capturing instructions include a face recognition instruction and a preview instruction, and the video display device sends the plurality of video capturing instructions to the video capturing device, so that the video capturing device generates a face recognition stream according to the face recognition instruction and generates a preview stream according to the preview instruction, and after the video capturing device splices the preview stream and the face recognition stream to obtain a data stream to be transmitted, the video capturing device sends the data stream to be transmitted to the video display device, and the user can see the preview stream and the face recognition stream through the video display device.
In an alternative implementation manner, the data stream to be transmitted may include channel information, where the channel information includes the number of channels in the second line and bandwidth information of each channel, and the channel information further includes the number of application video streams in the data stream to be transmitted and the channel where each application video stream is located. After the video display device receives the data stream to be transmitted, the information of each channel and the application video stream in the data stream to be transmitted is determined based on the channel information in the data stream to be transmitted, so that the video display device is prevented from splitting the data stream to be transmitted only according to the stream information of each application video stream, and the speed of the video display device for analyzing the data stream to be transmitted is improved. It should be understood that, in the embodiment of the present application, the bandwidth information included in the channel information may be characterized by the maximum resolution of the video stream that can be transmitted by the channel, and exemplary channel information includes that the bandwidth information of the channel 1 is 3840×540, the bandwidth information of the channel 2 is 3840×540, the bandwidth information of the channel 3 is 3840×240, which respectively indicates that the maximum resolution of the video stream that can be transmitted by the channel 1 may be 3840×540, the maximum resolution of the video stream that can be transmitted by the channel 2 may be 3840×540, and the maximum resolution of the video stream that can be transmitted by the channel 3 may be 3840×240.
In another optional implementation manner, the channel information includes resolution information of each application video stream, so that after the video display device obtains the channel information, computing resources can be allocated to playing of each application video stream according to the resolution information of each application video stream, the utilization rate of the computing resources of the video display device can be improved, and the video display device is prevented from being blocked; in the process of splitting the data stream to be transmitted by the video display device, the video display device not only can determine the position of the application video stream according to the channel where each application video stream is located, but also can determine the width and height information of the application video stream by utilizing the resolution information of the application video stream included in the channel information, thereby improving the accuracy of the video display device in analyzing the data stream to be transmitted.
In another alternative implementation, the channel information may be set in the header of the data stream to be transmitted. The video acquisition device sets the channel information at the head of the data stream to be transmitted, and sends the data stream to be transmitted to the video display device through the second line. In the prior art, the video display device can split the data stream to be transmitted only according to the stream information of each application video stream, which requires a large amount of computing resources, so that the video display device is blocked when playing the video stream.
In another alternative implementation, the data stream to be transmitted may further include stream information indicating attribute information of each application video stream in the data stream to be transmitted, the attribute information including at least one of exposure information, focus information, or white balance information of the application video stream. The video acquisition device sets the stream information in the data stream to be transmitted and sends the data stream to be transmitted to the video display device through the second line. After the video display device receives the data stream to be transmitted, the video display device can determine the number and the width and the height of each channel in the data stream to be transmitted and the number, the position and the resolution of the application video stream according to the channel information in the data stream to be transmitted, and further, the video display device can determine the attribute information of the application video stream according to the stream information in the data stream to be transmitted, so that identification of different types of application video streams is realized, and the accuracy of analyzing the data stream to be transmitted by the video display device is further improved.
In another optional implementation manner, the plurality of application video streams include a first application video stream and a second application video stream, and the resolution of the first application video stream and the resolution of the second application video stream are different. The embodiment of the application can simultaneously transmit a plurality of video streams with different resolutions, and the video acquisition device generates a plurality of application video streams by utilizing the original video streams, so that the video display device can be prevented from converting the original video streams, the computing resources of the video display device are released, and the video display device is prevented from playing the video streams and being blocked.
In another optional implementation manner, the data stream to be transmitted further includes a data area, where the data area includes a plurality of channels, and the video capturing device splices a plurality of application video streams to obtain the data stream to be transmitted, and includes: the video acquisition device respectively sets a plurality of application video streams in respective corresponding target channels, wherein the target channels corresponding to the application video streams are channels with the maximum transmission bandwidth being greater than or equal to the transmission bandwidth required by the application video streams. The video acquisition device takes a channel with the maximum transmission bandwidth being larger than or equal to the transmission bandwidth required by the application video stream as a target channel of the application video stream according to the bandwidth information of each channel in the data area and the resolution information of each application video stream in the channel information, and further, the video acquisition device sets the application video stream in the target channel corresponding to the application video stream, so that the application video stream is transmitted based on the transmission capacity of the channel. That is, each application video stream can be placed in a target channel with a transmission bandwidth greater than or equal to the transmission bandwidth required by the application video stream, so that the video acquisition device is prevented from placing the application video stream with larger resolution in a channel with smaller bandwidth information, and further, the transmission delay caused by insufficient transmission bandwidth of the channel is reduced.
In another optional implementation manner, the data stream to be transmitted includes a data area, the data area includes a plurality of channels, and the video acquisition device splices a plurality of application video streams to obtain the data stream to be transmitted, including: if the transmission bandwidth required by the first application video stream is greater than the transmission bandwidth of any one of the channels and is smaller than or equal to the maximum transmission bandwidth of the second line, the video acquisition device takes the first application video stream as a first data stream to be transmitted, and the first application video stream is any one of the application video streams; and if the transmission bandwidth required by the residual data stream is matched with the transmission bandwidths of the plurality of channels, the video acquisition device splices a plurality of application video streams in the residual data stream to obtain a second data stream to be transmitted, wherein the residual data stream comprises other application video streams except the first application video stream in the plurality of application video streams obtained according to the original video stream. The video acquisition device sends a data stream to be transmitted to the video display device through a second line, and the method comprises the following steps: the video acquisition device alternately sends the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second line.
In the prior art, after an application video stream with larger resolution is transmitted to a television, other application video streams are transmitted to the television, so that the transmission delay of other application video streams is higher. In other words, with the data stream transmission method provided by the embodiment of the application, the video acquisition device does not need to send the second data stream to be transmitted to the video display device after the transmission of the first data stream to be transmitted is finished, so that the defect that the video streams are independently transmitted (one video stream is transmitted after the transmission of the other video stream is finished) is avoided, and the transmission delay of the video streams is reduced.
In a second aspect, the present application provides a data streaming method, the method being performed by a video display apparatus, or the method being applicable to a terminal device that can support implementation of the method, e.g. the terminal device comprising a system-on-chip. In one possible design, the video capturing apparatus is connected to the video display apparatus through a first line and a second line, respectively, the first line being used for transmitting control instructions of the video capturing apparatus and the video display apparatus, the method comprising: the video display device receives the data stream to be transmitted sent by the video acquisition device through the second line, and also processes the data stream to be transmitted and plays at least one application video stream. The data stream to be transmitted is obtained by splicing a plurality of application video streams, and the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line. Therefore, the video display device transmits the control instruction through the first line, receives the data stream to be transmitted sent by the video acquisition device through the second line, realizes the separation transmission of the control instruction and the data stream, and avoids the problem that the control instruction occupies the bandwidth of the line for transmitting the data stream and causes higher transmission delay of the data stream when the control instruction and the data stream are transmitted through the same line. According to the data stream transmission method provided by the embodiment of the application, the data stream is transmitted by the exclusive second line, so that the transmission delay for transmitting the data stream to be transmitted is reduced, and the transmission rate for transmitting the data stream to be transmitted is improved; in addition, in the process that the video display device receives the data stream to be transmitted sent by the video acquisition device through the second line, the control instruction sent by the video acquisition device can be received through the first line, and the control instruction is sent to the video acquisition device without the video display device after the transmission process of the data stream to be transmitted is finished, so that the transmission delay of the control instruction is effectively reduced. In addition, compared with the prior art that only video streams can be transmitted independently (after one video stream is transmitted, the other video stream is transmitted), in the data stream transmission method used in the embodiment of the application, the video acquisition device can splice a plurality of application video streams to obtain the data stream to be transmitted, and the data stream to be transmitted is put into the second line for transmission, so that the video acquisition device does not need to transmit each application video stream independently, and the transmission delay of the video streams is reduced. In addition, since the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line, the video display device can transmit the data stream to be transmitted according to the maximum transmission bandwidth of the second line, thereby avoiding transmission delay caused by insufficient transmission bandwidth of the second line.
In addition, since the data stream to be transmitted includes a plurality of application video streams, after the video display device receives the data stream to be transmitted, the video display device can process and play the data stream to be transmitted, and a user can view various video information according to the original video stream through the video display device, for example, the user can view preview stream and photographing stream in video call, and view face type identification information (for example, classifying face types of faces) obtained according to the original video stream in the process of video call, so that viewing experience of the user is improved.
In one possible example, the data stream to be transmitted includes channel information including the number of channels in the second line and bandwidth information of each channel, and the channel information further includes the number of application video streams in the data stream to be transmitted and the channel in which each application video stream is located. After the video display device receives the data stream to be transmitted, the video display device can determine the information of each channel and the application video stream in the data stream to be transmitted based on the channel information in the data stream to be transmitted, so that the video display device is prevented from splitting the data stream to be transmitted only according to the stream information of each application video stream, and the speed of the video display device for analyzing the data stream to be transmitted is improved.
In another possible example, the channel information includes resolution information of each application video stream. After the video display device acquires the channel information, computing resources can be allocated for playing the application video stream according to the resolution information of each application video stream, so that the utilization rate of the computing resources of the video display device can be improved, and the video display device is prevented from being blocked; in the process of splitting the data stream to be transmitted by the video display device, the video display device not only can determine the position of the application video stream according to the channel where each application video stream is located, but also can determine the width and height information of the application video stream by utilizing the resolution information of the application video stream included in the channel information, thereby improving the accuracy of the video display device in analyzing the data stream to be transmitted.
In another possible example, the channel information is provided at the head of the data stream to be transmitted. In the prior art, the video display device can split the data stream to be transmitted only according to the stream information of each application video stream, which requires a large amount of computing resources to cause the video display device to play the video stream to be blocked.
In an alternative implementation, the video display apparatus processes a data stream to be transmitted and plays at least one application video stream, including: the video display device splits the data stream to be transmitted according to the channel information to obtain all application video streams; the video display device selects at least one application video stream from all application video streams to play. After the video display device receives the data stream to be transmitted, the video display device can determine the number and the width and the height of each channel in the data stream to be transmitted and the number, the position and the resolution of the application video stream according to the channel information in the data stream to be transmitted, and further, the video display device can determine the attribute information of the application video stream according to the stream information in the data stream to be transmitted, so that identification of different types of application video streams is realized, and the accuracy of analyzing the data stream to be transmitted by the video display device is improved.
In one possible example, the application video stream played by the video display device is: the video display device splits the data stream to be transmitted according to the channel information, and performs AI processing on the application video stream obtained by splitting to obtain an AI data stream, where the AI processing may be, but is not limited to: target detection, face recognition, object classification, color recognition, object counting, etc. For example, the video display device includes a processor, where the processor may be configured to perform an operation corresponding to the above-mentioned AI process, and display a result of the operation on a display screen of the video display device, so that a user may see, through the display screen of the video display device, various video information of an original video stream, such as a preview stream and a photo stream in a video call, and view, during the video call, face type identification information (such as classifying a face type of a face) obtained according to the original video stream, thereby improving a user experience of viewing the video.
In another optional implementation manner, before the video display apparatus receives the data stream to be transmitted sent by the video acquisition apparatus through the second line, the data stream transmission method may further include: the video display device generates at least one acquisition instruction in response to the operation information acting on the video display device. The video display device can send the acquisition instruction to the video acquisition device, so that the video acquisition device generates a plurality of application video streams according to the acquisition instruction, and the data streams to be transmitted obtained by splicing the application video streams are sent to the video display device through the second line, so that a user can see various video information (such as target detection information, face and face classification, color recognition and the like) of the original video stream through a display screen of the video display device, and the experience of watching the video by the user is improved.
In a third aspect, embodiments of the present application provide a video capturing apparatus, which may refer to the description of any one of the first aspect, and will not be described herein. The video capture device has functionality to implement the actions in the method examples of any of the above-described first aspects. The functions may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. In one possible design, the video capturing apparatus is connected to the video display apparatus through a first line and a second line, respectively, the first line being used for transmitting control instructions of the video capturing apparatus and the video display apparatus, the video capturing apparatus comprising: the processing unit is used for generating a plurality of application video streams according to the original video streams acquired by the video acquisition device; the splicing unit is used for splicing the plurality of application video streams to obtain a data stream to be transmitted, and the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line; and the receiving and transmitting unit is used for sending the data stream to be transmitted to the video display device through the second line.
In an alternative implementation, the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, and the channel information further includes the number of application video streams in the data stream to be transmitted, and the channel in which each application video stream is located.
In another alternative implementation, the channel information includes resolution information for each application video stream.
In another alternative implementation, the channel information is provided at the head of the data stream to be transmitted.
In another alternative implementation, the plurality of application video streams includes a first application video stream and a second application video stream, the first application video stream having a different resolution than the second application video stream.
In another optional implementation manner, the data stream to be transmitted further includes a data area, where the data area includes a plurality of channels, and the splicing unit is specifically configured to set the plurality of application video streams in respective corresponding target channels respectively; the target channel corresponding to the application video stream is a channel with the maximum transmission bandwidth being greater than or equal to the transmission bandwidth required by the application video stream.
In another optional implementation manner, the data stream to be transmitted includes a data area, where the data area includes a plurality of channels, and the splicing unit is specifically configured to take the first application video stream as the first data stream to be transmitted if a transmission bandwidth required by the first application video stream is greater than a transmission bandwidth of any one of the plurality of channels and less than or equal to a maximum transmission bandwidth of the second line, where the first application video stream is any one of the plurality of application video streams; the splicing unit is specifically configured to splice multiple application video streams in the remaining data stream to obtain a second data stream to be transmitted if the transmission bandwidth required by the remaining data stream is matched with the transmission bandwidths of the multiple channels, where the remaining data stream includes application video streams other than the first application video stream among the multiple application video streams obtained according to the original video stream; the transceiving unit is specifically configured to alternately send the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second line.
In another alternative implementation, the plurality of application video streams includes at least one of a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitor stream, a time-of-flight ranging stream, or a gesture control stream.
In another alternative implementation, the first line is a USB and the second line is an HDMI line.
In a fourth aspect, embodiments of the present application provide a video display apparatus, and the beneficial effects may be seen in the description of any one of the second aspects, which is not repeated here. The video capture device has the functionality to implement the behavior in the method example of any of the second aspects described above. The functions may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. In one possible design, the video display device is connected to the video capture device through a first line and a second line, respectively, the first line being used to transmit control instructions for the video capture device and the video display device, the video display device comprising: the receiving and transmitting unit is used for receiving a data stream to be transmitted sent by the video acquisition device through a second line, wherein the data stream to be transmitted is obtained by splicing a plurality of application video streams, and the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line; the processing unit is used for processing the data stream to be transmitted to obtain a plurality of application video streams; and the display unit is used for playing at least one application video stream in the plurality of application video streams.
In an alternative implementation, the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, and the channel information further includes the number of application video streams in the data stream to be transmitted, and the channel in which each application video stream is located.
In another alternative implementation, the channel information includes resolution information for each application video stream.
In another alternative implementation, the channel information is provided at the head of the data stream to be transmitted.
In another optional implementation manner, the processing unit is specifically configured to split the data stream to be transmitted according to the channel information to obtain all application video streams; the display unit is specifically configured to select at least one application video stream from all application video streams for playing.
In a fifth aspect, embodiments of the present application provide a terminal device, including a processor and an interface circuit, where the processor receives or transmits data through the interface circuit, and the processor is configured to implement the method in any one of the possible implementations of the first aspect and the first aspect, or the operation steps of the method in any one of the possible implementations of the second aspect and the second aspect through logic circuits or executing code instructions.
In a sixth aspect, embodiments of the present application provide a computer readable storage medium, in which a computer program or instructions is stored which, when executed by a terminal device, implement the method in any one of the possible implementations of the first aspect and the first aspect, or the operation steps of the method in any one of the possible implementations of the second aspect and the second aspect.
In a seventh aspect, embodiments of the present application provide a computer program product comprising instructions which, when run on a terminal device or a processor, cause the terminal device or the processor to execute the instructions to implement the method in any one of the possible implementations of the first aspect and the first aspect, or the operational steps of the method in any one of the possible implementations of the second aspect and the second aspect.
In an eighth aspect, embodiments of the present application provide a chip, including a memory for storing computer instructions, and a processor for calling and executing the computer instructions from the memory to perform the method of any of the possible implementations of the first aspect and the first aspect, or the operation steps of the method of any of the possible implementations of the second aspect and the second aspect.
Further combinations of the present application may be made to provide further implementations based on the implementations provided in the above aspects.
Fig. 1 is a schematic diagram of a video communication system according to the present application;
fig. 2 is a schematic flow chart of a data stream transmission method according to the present application;
FIG. 3 is a schematic diagram of an intelligent screen according to the present application;
fig. 4 is a schematic structural diagram of a data stream to be transmitted according to the present application;
fig. 5 is a schematic flow chart of another data stream transmission method according to the present application;
FIG. 6 is a schematic diagram of an alternate transmission of data streams according to the present application;
fig. 7 is a schematic diagram of a data stream transmission system according to the present application;
fig. 8 is a schematic structural diagram of a terminal device according to the present application.
The terms first, second, third and the like in the description and in the claims and in the above-described figures, are used for distinguishing between different objects and not necessarily for limiting a particular order.
In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
For clarity and conciseness in the description of the embodiments below, a brief description of the related art will be given first.
In the current technical scheme, the camera transmits the video stream between the camera and the television through the USB, and the transmission capacity of the USB is smaller, so that the USB has higher time delay when transmitting the video stream. For example, the transmission bandwidth required for video streaming of three-dimensional (3D) images is about 442MB/s, whereas the transmission bandwidth of USB3.0 can generally only reach 400MB/s, resulting in a higher latency for USB in transmitting video streams.
In order to solve the above-mentioned problems, the present application provides a data stream transmission method, in which a video capturing device is connected to a video display device through a first line and a second line, respectively, and the first line is used for transmitting control instructions of the video capturing device and the video display device, the method includes: the video acquisition device is used for acquiring an original video stream, generating a plurality of application video streams according to the original video stream, splicing the application video streams to obtain a data stream to be transmitted, and further sending the data stream to be transmitted to the video display device through a second line; the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line. The video display device is used for processing the data stream to be transmitted and playing at least one display video stream after receiving the data stream to be transmitted. Therefore, the video acquisition device transmits the control instruction through the first line, and the data stream to be transmitted is sent to the video display device through the second line, so that the separation transmission of the control instruction and the data stream is realized, and the problem that the transmission delay of the data stream is higher because the control instruction occupies the bandwidth of the line for transmitting the data stream when the control instruction and the data stream are transmitted through the same line is avoided.
According to the data stream transmission method provided by the embodiment of the application, the data stream is transmitted by the exclusive second line, so that the transmission delay for transmitting the data stream to be transmitted is reduced, and the transmission rate for transmitting the data stream to be transmitted is improved; in addition, in the process that the video acquisition device sends the data stream to be transmitted to the video display device through the second line, the control instruction sent by the video display device can be received through the first line, and the video display device does not need to send the control instruction to the video acquisition device after the transmission process of the data stream to be transmitted is finished, so that the transmission time delay of the control instruction is effectively reduced.
In addition, compared with the prior art that only video streams can be transmitted independently (after one video stream is transmitted, the other video stream is transmitted), in the data stream transmission method used in the embodiment of the application, the video acquisition device can splice a plurality of application video streams to obtain the data stream to be transmitted, and the data stream to be transmitted is put into the second line for transmission, so that the video acquisition device does not need to transmit each application video stream independently, and the transmission delay of the video streams is reduced. In addition, since the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line, the video acquisition device can transmit the data stream to be transmitted according to the maximum transmission bandwidth of the second line, thereby avoiding transmission delay caused by insufficient transmission bandwidth of the second line.
The video capture device may be a camera or video camera or the like, the camera being used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element, thereby generating an original video stream. The photosensitive element may be a charge coupled device (Charge Coupled Device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive elements convert the optical signals to electrical signals, which are then transferred to a graphics processor (Graphics Processing Unit, GPU) or an image signal processor (Image Signal Processor, ISP) for conversion to digital image signals.
The video display device is used for displaying images, videos, etc., and may be a television, a projector, a computer monitor, etc. In some possible embodiments, the video display device may be other terminal equipment with image display function, such as a wearable device, an in-vehicle device, an augmented Reality (Augmented Reality, AR)/Virtual Reality (VR) device, a notebook computer, an Ultra-Mobile Personal Computer (UMPC), a netbook, a personal digital assistant (Personal Digital Assistant, PDA), etc.
By way of example, fig. 1 is a schematic diagram of a video communication system provided by the present application that includes a television 110, a retractable camera 120, a video camera 130, and a conference tablet 140. The television 110 and conference tablet may communicate over a network 150. Alternatively, the network 150 may include switches, servers, and the like.
As an alternative implementation, the video capturing apparatus and the video display apparatus may be integrated on one terminal device, and as shown in fig. 1, the retractable camera 120 is disposed on the television 110, and the device integrated with the television function and the camera function may be referred to as a smart screen.
In one possible example, multiple video capturing devices may be provided on the video display device, e.g., multiple cameras may be provided on the television 110, and other capturing devices may also be interfaced through the interface.
As another alternative implementation, the video capturing device and the video display device may be separately provided, as shown in fig. 1, where the video capturing device may be a camera 130 and the video display device may be a conference tablet 140.
The video display device may include 1 or more display screens for displaying images, videos, and the like. The display screen includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED) or an active-matrix organic light-emitting diode (matrix organic light emitting diode), a flexible light-emitting diode (flex), a Mini-LED, a Micro-OLED, a quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), or the like.
The video display device may further include a processor, for example: the processors may include application processors (Application Processor, APs), modem processors, GPUs, ISPs, controllers, memories, video codecs, digital signal processors (Digital Signal Processor, DSPs), baseband processors, and/or Neural network processors (Neural-Network Processing Unit, NPUs), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors. The controller may be a neural center pivot and command center of the video display device. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.
A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may hold instructions or data that the processor has just used or recycled. If the processor needs to reuse the instruction or data, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor is reduced, so that the data stream transmission efficiency is improved.
In some embodiments, the processor may include one or more interfaces. The interfaces may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, a USB interface, and/or an HDMI interface, among others. The USB interface is an interface conforming to the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. In the embodiment of the application, the USB interface is used for transmitting the control instruction.
As shown in fig. 1, the USB interface may be used to connect to a charger to supply power to the television 110, or may be used to transmit control commands between the television 110 and peripheral devices. And can also be used for connecting with a headset, and playing audio through the headset. The USB interface may also be used to connect other terminal devices, such as AR devices, retractable cameras 120, etc.
It should be noted that the video communication scenario shown in fig. 1 is only one possible implementation manner provided by the embodiment of the present application, and the data stream transmission method provided by the present application is used in image transmission links, such as fields related to image transmission, such as photography, camera shooting, and monitoring.
The following describes in detail the implementation of the embodiment of the present application with reference to the drawings.
As shown in fig. 2, fig. 2 is a schematic flow chart of a data stream transmission method provided by the present application, and a video acquisition device 210 is connected to a video display device 220 through a first line and a second line, respectively. Illustratively, the scalable camera 120 is a video capturing device 210, and the television 110 is a video display device 220, and the data stream transmission method provided by the present application is described, where the video capturing device 210 may implement the functions of the scalable camera 120 shown in fig. 1, and the video display device 220 may implement the functions of the television 110 shown in fig. 1.
The first circuit is used for transmitting control instructions of the video acquisition device 210 and the video display device 220, the second circuit is used for transmitting data streams to the video acquisition device 210 by the video acquisition device 220, so that separate transmission of the data streams and the control instructions is realized, in the process that the video acquisition device 210 transmits the data streams to be transmitted to the video display device 220 through the second circuit, the video display device 220 can transmit the control instructions to the video acquisition device 210 through the first circuit, so that the video display device 220 does not need to transmit the control instructions to the video acquisition device 210 after the transmission process of the data streams to be transmitted is finished, and the transmission delay of the control instructions is reduced.
With continued reference to fig. 2, the data stream transmission method provided in the embodiment of the present application may include the following steps.
S210, the video display apparatus 210 generates an acquisition instruction in response to the operation information acting on the video display apparatus.
Taking the video display device 220 with the video capturing device 210 as an example, as shown in fig. 3, fig. 3 is a schematic diagram of a smart screen provided by the present application, the video display device 220 may be a smart screen 300, and the video capturing device 210 is a camera 310 on the smart screen 300.
For example, the operation information may be a remote control instruction received by the smart screen 300, which the user transmits through the remote controller of the smart screen 300.
As another example, the operation information is a control command sent by a remote device (e.g., a mobile phone) communicatively connected to the smart screen 300 through an Application (APP).
As another example, the operation information is voice information received by an audio input device of the smart screen 300, which may be a microphone provided on the smart screen 300, and which may be a microphone provided on a remote controller of the smart screen 300.
For another example, the operation information may also be gesture information of the user received by the smart screen 300, and if the display screen 320 of the smart screen 300 is a touch display screen, the gesture information may be an operation performed by the user on the touch display screen; the gesture information may be a gesture operation in which the user performs a non-touch operation (e.g., a blank operation) according to a predetermined motion.
The above operation information is merely a possible implementation manner provided by the present application, and is not meant to limit the present application. In some possible examples, the above-mentioned operation information may also be an instruction transmitted from a server connected to the video display apparatus by wire.
S220, the video display device 220 sends an acquisition instruction to the video acquisition device 210 through the first line.
In one possible example, the first line may be a USB. For example, the processor of the video capturing apparatus 210 has a USB interface, the processor of the video display apparatus 220 also has a USB interface, and two ends of the first line are respectively connected to the 2 USB interfaces.
In another possible example, the first line may also be implemented through other interfaces of the video capture device 210 and the video display device 220, such as, but not limited to: mini USB interface, micro USB interface, USB Type C interface, etc.
The control instruction between the video acquisition device and the video display device is transmitted by utilizing the first line, so that the video stream and the control instruction are prevented from being transmitted in the same line, and further, the transmission time delay of the control instruction opposite to the transmission direction of the video stream in the unidirectional transmission process of the video stream is reduced.
The video capturing device 210 generates a plurality of application video streams according to the original video streams captured by the video capturing device 210 at S230.
For example, the original video stream is a source of video data obtained by the video capture device 210 capturing a target object or person.
In one possible example, the plurality of application video streams may be a plurality of video streams with the same resolution, for example, each application video stream is a video stream of "1080p@60," 1080P "refers to the resolution information of the video stream being 1920×1080 (the width is 1920, the height is 1080), and" @60 "refers to the frame rate information of the video stream being 60 frames/second.
In another possible example, the plurality of application video streams may also be a plurality of video streams with different resolutions, and the plurality of application video streams includes, for example, a first application video stream and a second application video stream, where the resolution of the first application video stream and the resolution of the second application video stream are different. For example, the resolution information of the first application video stream is 1080P (width 1920, height 1080), and the resolution information of the second application video stream is 720P (width 1080, height 720).
The generation of the plurality of application video streams by the video capturing device 210 may be implemented based on a plurality of capturing instructions, for example, the video display device 220 generates a plurality of capturing instructions according to gesture information of a user, where the plurality of capturing instructions include a face recognition instruction and a preview instruction, and the video display device 220 sends the plurality of video capturing instructions to the video capturing device 210, so that the video capturing device 210 generates a face recognition stream according to the face recognition instruction and generates a preview stream according to the preview instruction, and after the video capturing device 210 splices the preview stream with the face recognition stream to obtain a data stream to be transmitted, the video capturing device 210 sends the data stream to be transmitted to the video display device 220, and the user can see the preview stream and the face recognition stream through the video display device 220. Because the plurality of application video streams are generated by the video acquisition device 210 according to the original video streams, the video display device 220 is prevented from converting the original video streams, and the computing resources of the video display device 220 are released, so that the video display device 220 plays the video streams without being blocked, and the video watching experience of users is improved.
As an alternative implementation, the plurality of application video streams comprised by the data stream to be transmitted may be heterogeneous video streams. For example, the plurality of application video streams includes at least one video stream of a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitor stream, a time-of-flight ranging stream, or a gesture control stream. The user can view different video information of the original video stream by using the video display device, for example, the user can view the preview stream and the photographing stream in the video call, and view face recognition information (for example, classifying the face of the user) obtained according to the original video stream in the video call process, so that the user experience of viewing the video is improved.
The preview stream refers to a video stream of the video acquisition device for previewing the acquired video data source. Such as the preview stream shown in fig. 3.
The photographing flow refers to a video flow obtained by photographing a target object or person by the video acquisition device. For example, a photographing flow shown in fig. 3.
A depth data stream refers to a set of image streams having as pixel values the distance (depth) from a video acquisition device to points in a scene, which directly reflects the geometry of the visible surface of the scene. The depth image data can be calculated as point cloud data through coordinate conversion, and the point cloud data with regular and necessary information can also be reversely calculated as the depth image data.
The face stream refers to a face information stream in an original video stream shot by the video acquisition device.
The target detection flow refers to a data flow which is obtained after the video acquisition device processes the original video flow and is matched with the target detection requirement. For example, the target detection requirement may be that the object category in the original video stream is a specified category, the clothing color of the person is a specified color, and so on. As shown in fig. 3, the target detection stream may be a bone identification of a person in the original video stream, so as to obtain bone data of the person, and the smart screen 300 may display the bone data stream.
The monitoring stream is a video stream obtained by the video acquisition device for recording the monitoring scene. Such as the monitor flow shown in fig. 3.
The time-of-flight ranging flow is a data flow generated by the video acquisition device according to the distance between each point on the target object by continuously transmitting light pulses to the target object and then receiving light returned from the object by the sensor and obtaining the distance between the video acquisition device and the target object by detecting the flight (round trip) time of the light pulses.
The gesture control flow is a data flow obtained by the video acquisition device according to gesture information of a target person in the original video flow. For example, the gesture control flow shown in fig. 3.
It is noted that fig. 3 is only a possible implementation provided by the present application, and should not be construed as limiting the present application. In some possible embodiments, the video display device may also display more or fewer video streams.
Each application video stream can be different types of video streams, a user can view different video information according to the original video streams by utilizing the video display device, for example, the user can view preview streams and photographing streams in video call, and view face type identification information (for example, classifying face types of faces) obtained according to the original video streams in the video call process, so that the user experience of viewing videos is improved.
As an alternative implementation, the video capture device may generate the plurality of application video streams based on a plurality of capture instructions. As shown in fig. 3, for example, a user may click on a video call to generate a preview stream and a photo stream; for another example, the user can generate a face recognition instruction through gesture operation, and then the video acquisition device generates a face stream according to the original video stream; for another example, the user sends out a body-building posture identification instruction through the remote controller, and the video acquisition device generates a depth data stream (such as a bone data stream shown in fig. 3) according to the original video stream. The above examples are implemented by different instructions through different operation information, and in some possible examples, a user may perform a plurality of operations within a period of time, and the video capture device may generate a capture instruction according to the plurality of operations, where the capture instruction may instruct the video capture device to generate an application video stream corresponding to the plurality of operations.
With reference to fig. 2, the data stream transmission method provided by the present application further includes the following steps.
S240, the video acquisition device 210 splices the application video streams to obtain a data stream to be transmitted.
The data stream to be transmitted may include channel information, which may include the number of application video streams in the data stream to be transmitted. By way of example, as shown in fig. 4 (a), the plurality of application video streams includes application video stream 1 to application video stream 4; as shown in fig. 4 (c), the channel information indicates that the data stream to be transmitted includes 4 application video streams, where the 4 application video streams are application video streams 1 to 4, and the video capture device 210 may splice the application video streams 1 to 4 to obtain the data stream to be transmitted.
In one possible example, the channel information may also include the channel in which each application video stream is located. By way of example, as shown in fig. 4 (c), an application video stream 1 is set in a channel 1.
The channel information may also include the number of channels in the second line, as well as bandwidth information for each channel. The bandwidth information refers to the width and height of a channel, and the bandwidth information of the channel determines the maximum resolution of the application video stream that the channel can transmit. As shown in fig. 4 (b), the data stream to be transmitted includes an information area 1, an information area 2, and a data area, where the data area includes a plurality of channels (channels 1-6), for example, the bandwidth information of channels 1 and 2 is "3840×540", the bandwidth information of channels 3-6 is "3840×240", the maximum resolution of the application video streams that can be transmitted by channels 1 and 2 is 1080P (1920×1080), and the maximum resolution of the application video streams that can be transmitted by channels 3-6 is 720P (1280×720).
As an optional implementation manner, S240 specifically includes: the video capture device 210 sets a plurality of application video streams in respective corresponding target channels. The target channel corresponding to the application video stream is a channel with the maximum transmission bandwidth being greater than or equal to the transmission bandwidth required by the application video stream. The transmission bandwidth required by the application video streams is determined by the video acquisition device according to the resolution information of the application video streams, each application video stream can be placed in a target channel with the transmission bandwidth larger than or equal to the transmission bandwidth required by the application video streams, the situation that the video acquisition device places the application video stream with larger resolution in a channel with smaller bandwidth information is avoided, and then the transmission delay caused by insufficient transmission bandwidth of the channel is reduced.
Wherein the transmission bandwidth of the channel indicates the maximum amount of data of the video stream that the channel can transmit per unit time. Here, taking an example in which N frames of images with the same resolution are required to be transmitted by the video stream in a unit time, the maximum data amount=resolution of a single frame image×n, N is a positive integer. The transmission bandwidth required for the application video stream described above indicates the amount of data to be transmitted for all image frames in the application video stream per unit time.
In an alternative implementation, the channel information may also include resolution information for each application video stream. For example, the resolution information of the application video streams 1 to 4 and the bandwidth information of the channels 1 to 6 shown in fig. 4 are shown in table 1.
TABLE 1
| Channel | Bandwidth of a communication device | Video streaming | Resolution ratio |
| Channel 1 | 3840*540 | Application video stream 1 | 1080P(1920*1080) |
| Channel 2 | 3840*540 | Application videoStream 2 | 1600*1200 |
| Channel 3 | 3840*240 | Application video stream 3 | 720P(1280*720) |
| Channel 4 | 3840*240 | Application video stream 4 | 540P(960*540) |
| Channel 5 | 3840*240 | ||
| Channel 6 | 3840*240 |
Here, taking the same number of image frames transmitted in unit time by different application video streams as an example, the comparison of transmission bandwidths is described by taking resolution as an example, the maximum transmission bandwidth of the channel 1 and the channel 2 is 3840×540, and the maximum transmission bandwidth indicates that the maximum data volume that can be transmitted by the channel 1 and the channel 2 is a video stream with the resolution of 1080P (1920×1080). As shown in fig. 4, since the transmission bandwidth required for the application video stream 1 (resolution 1080P) and the application video stream 2 (resolution 1600 x 1200) is less than or equal to the maximum transmission bandwidths of the channels 1 and 2, the application video stream 1 may be placed in the channel 1 and the application video stream 2 may be placed in the channel 2.
The maximum transmission bandwidth of channels 3-6 is 3840×240, which indicates that the maximum amount of data that channels 3-6 can transmit is a video stream with a resolution of 720P (1280×720). As shown in fig. 4, since the transmission bandwidth required for the application video stream 3 (resolution 720P) and the application video stream 4 (resolution 540P) is less than or equal to the maximum transmission bandwidths of the channels 3 to 6, the application video stream 3 can be placed in the channel 3 and the application video stream 4 can be placed in the channel 4.
In a first possible example, the video capturing apparatus 210 respectively sets a plurality of application video streams in respective corresponding target channels specifically includes: the video capture device 210 determines a transmission bandwidth required by the application video stream according to the resolution information of the application video stream, and places the application video stream in a target channel with a transmission bandwidth greater than or equal to the transmission bandwidth required by the application video stream. The video acquisition device can determine the target channel based on the resolution information of the application video stream, so that the application video stream with larger resolution is prevented from being put into the channel with smaller bandwidth information, and the transmission delay caused by insufficient transmission bandwidth of the channel is reduced.
In a second possible example, the plurality of application video streams includes a first application video stream and a second application video stream, and the video capturing device 210 sets the plurality of application video streams in respective corresponding target channels specifically includes: the video acquisition device determines a first transmission bandwidth according to the resolution information of the first application video stream, and determines a second transmission bandwidth according to the resolution information of the second application video stream, and further, the video acquisition device sets the first application video stream and the second application video stream in the target channel, wherein the sum of the first transmission bandwidth and the second transmission bandwidth is smaller than or equal to the maximum transmission bandwidth of the target channel. In connection with the contents shown in table 1 and fig. 4, assuming that the first application video stream may be the application video stream 3, the second application video stream may be the application video stream 4, the target channel may be the channel 1, and since the total transmission bandwidth (1280×720+960×540=3840×375) required for the application video stream 3 and the application video stream 4 is smaller than the maximum transmission bandwidth (3840×540) of the channel 1, the video capture device 210 may put the application video stream 3 and the application video stream 4 into the channel 1. The video acquisition device can put a plurality of application video streams into the same target channel, and the total transmission bandwidth required by the application video streams is smaller than or equal to the maximum transmission bandwidth of the target channel, so that the transmission capacity of each channel in the second line can be effectively utilized, and the waste of the transmission capacity of the second line is avoided.
The video acquisition device takes a channel with the maximum transmission bandwidth being larger than or equal to the transmission bandwidth required by the application video stream as a target channel of the application video stream according to the bandwidth information of each channel in the data area and the resolution information of each application video stream in the channel information, and further, the video acquisition device sets the application video stream in the target channel corresponding to the application video stream, so that the application video stream is transmitted based on the transmission capacity of the channel. That is, each application video stream can be placed in a target channel with a transmission bandwidth greater than or equal to the transmission bandwidth required by the application video stream, so that the video acquisition device is prevented from placing the application video stream with larger resolution in a channel with smaller bandwidth information, and the transmission delay caused by insufficient transmission bandwidth of the channel is further reduced.
The channel information provided by the present application may be adjusted according to application scenarios of the video capturing apparatus and the video display apparatus, where the application scenarios of the video capturing apparatus and the video display apparatus are different, and table 1 is merely an example provided for illustrating a data stream transmission method according to an embodiment of the present application, and should not be construed as limiting the present application. For example, the number of channels and the bandwidth information in the channel information may be preconfigured according to application scenarios of the video capturing apparatus and the video display apparatus, for example, in a video call scenario, the data area may include 2 channels, and the 2 channels are used for transmitting the preview stream and the photo stream, respectively. As another example, the number of channels in the channel information and the bandwidth information may be dynamically generated according to the number of application video streams and the resolution information.
S250, the video acquisition device sends the data stream to be transmitted to the video display device through the second line.
The transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line. As shown in fig. 4, the total transmission bandwidth required for the application video stream 1 to the application video stream 4 is smaller than the maximum transmission bandwidth of the second line.
The maximum transmission bandwidth of the second line indicates the maximum data amount of the video stream that the second line can transmit in a unit time; the transmission bandwidth required for the data stream to be transmitted indicates the amount of data to be transmitted in a unit time, where the description information may be used to describe resolution information, exposure information, white balance information, or focusing information of the application video stream, and the description information may include the channel information.
In one possible example, the second line may be HDMI, or VGA (video graphics array), a digital video interface (digital visual interface, DVI) line, or a display interface (DP) line, or the like.
For example, if the second line is HDMI, the format of the application video stream may be YUV format, where Y information indicates brightness (luminence) of each frame image in the application video stream, and U and V information indicates color information (chromance) of each frame image in the application video stream.
Because the video stream in YUV format is a lossless video stream, the transmission bandwidth of USB in the prior art is smaller, resulting in higher transmission delay in YUV format. According to the data stream transmission method provided by the embodiment of the application, the second line is the HDMI line, and the transmission bandwidth of the HDMI line is larger than that of the USB, so that the transmission time delay of the video acquisition device for sending the data stream to be transmitted to the video display device can be reduced.
In addition, in the prior art, in the process of transmitting a plurality of application video streams by using a USB, the camera encodes the application video streams, so as to compress the application video streams to obtain a data stream to be transmitted, wherein the data stream to be transmitted comprises encoding information, and the encoding information indicates the encoding mode of the camera on the application video streams; because the video stream in the data stream to be transmitted is compressed, the video stream included in the data stream to be transmitted has a loss compared with the application video stream, and further, the video stream played on the television is not completely consistent with the application video stream, thereby reducing the playing effect of the video stream. In the data stream transmission method provided by the embodiment of the application, under the condition that the second line is an HDMI line and the format of the application video stream is a YUV format, the video acquisition device does not need to encode the application video stream, and no encoding information is generated, so that fields for describing the format of the application video stream in the data stream to be transmitted are reduced, redundancy of the data stream to be transmitted is reduced, the data amount of the data stream to be transmitted is reduced, the video acquisition device sends the data stream to be transmitted to the video display device by using HDMI, and further, the speed for transmitting the data stream to be transmitted is improved, and the transmission time delay of the data stream to be transmitted is reduced. In addition, the video acquisition device can transmit the data stream in YUV format by using the exclusive second line, so that no difference exists between the video stream played by the video display device and the application video stream generated by the video acquisition device, the effect of playing the video stream by the video display device is improved, and the experience of watching the video by a user is further improved.
It should be noted that the format of the application video stream is YUV format, which is only one possible example provided by the embodiments of the present application, and in some possible examples, the format of the application video stream may be lossless RGB (red green blue) format, and the format of the application video stream may also be lossy video format, such as audio-video interleaving format (audio video interleaved, AVI) or video file format (real audio variable bit, RMVB), and so on.
As an alternative implementation, the channel information may be provided at the head of the data stream to be transmitted. As shown in fig. 4, channel information (inner metadata) may be provided in the information area 1.
The video acquisition device sets the channel information at the head of the data stream to be transmitted, and sends the data stream to be transmitted to the video display device through the second line. In the prior art, the video display device can split the data stream to be transmitted only according to the stream information of each application video stream, which requires a large amount of computing resources to cause the video display device to play the video stream to be blocked.
As another alternative implementation manner, the data stream to be transmitted may further include stream information indicating attribute information of each application video stream in the data stream to be transmitted, the attribute information including at least one of exposure information, focus information, or white balance information of the application video stream. By way of example, the attribute information may be obtained by the video capture device using 3A (3-automatic) digital imaging technology, and the attribute information may include automatic exposure (automatic exposure, AE) information, automatic Focus (AF) information, and automatic white balance (automatic white balance, AWB) information. For the relevant content of the 3A digital imaging technology, reference may be made to the relevant description of the prior art, and the description is omitted here.
In some possible examples, as shown in fig. 4, stream information may be provided in the information area 2. It should be noted that the information area 2 and the data area shown in fig. 4 are only examples given for illustrating the above embodiments of the present application, and in some possible examples, the information area 2 may be disposed after the data area or between any 2 channels.
The video acquisition device sets the stream information in the data stream to be transmitted and sends the data stream to be transmitted to the video display device through the second line. After the video display device receives the data stream to be transmitted, the video display device can determine the number and the width and the height of each channel in the data stream to be transmitted and the number, the position and the resolution of the application video stream according to the channel information in the data stream to be transmitted, and further, the video display device can determine the attribute information of the application video stream according to the stream information in the data stream to be transmitted, so that identification of different types of application video streams is realized, and the accuracy of analyzing the data stream to be transmitted by the video display device is improved.
In one possible embodiment, the plurality of application video streams may not be identical, and there may be a video stream in the plurality of application video streams that is greater than the maximum transmission bandwidth of the channel, which may result in failure of any one of the channels in the second line to transmit the video stream. In order to solve the above-mentioned problem, the present application provides a possible implementation manner, as shown in fig. 5, fig. 5 is a schematic flow chart of another data stream transmission method provided in the present application, and S240 specifically includes S2401 and S2402.
In S2401, if the transmission bandwidth required by the first application video stream is greater than the transmission bandwidth of any one of the channels and less than or equal to the maximum transmission bandwidth of the second line, the video capture device 210 uses the first application video stream as the first data stream to be transmitted.
The first application video stream is any one of a plurality of application video streams. For example, in combination with the contents shown in fig. 4 and table 1, if the resolution information of the first application video stream is 2K (2560×1440) and the maximum transmission bandwidth in channels 1 to 6 is 3840×540 < 2K, the video capturing device 210 uses the first application video stream as the first data stream to be transmitted.
S2402, if the transmission bandwidth required by the residual data stream matches the transmission bandwidths of the channels, the video capture device 210 splices the application video streams in the residual data stream to obtain a second data stream to be transmitted.
The remaining data stream includes other application video streams except the first application video stream from the multiple application video streams obtained by the video capture device 210 according to the original video stream. For a related example of the second data stream to be transmitted, reference may be made to the above description of S240, which is not repeated here.
With continued reference to fig. 5, S250 specifically includes S2501.
S2501, the video capturing apparatus 210 alternately sends the first data stream to be transmitted and the second data stream to be transmitted to the video display apparatus 220 through the second line.
Fig. 6 is a schematic diagram of alternate transmission of a data stream according to the present application, where the second line is an HDMI line, the maximum transmission bandwidth of the HDMI line may support transmission of a video stream of 4k@60 (resolution 4096×2160, frame rate 60 frames/second), the first application video stream is 4k@30 (resolution 4096×2160, frame rate 30 frames/second), and the remaining data stream includes 4 application video streams (application video streams 1 to 4 shown in table 1, each having a frame rate of 30 frames/second), and the video capturing device 210 and the video display device 220 may alternately transmit the first data stream to be transmitted and the second data stream to be transmitted in frame units. For example, the video capturing device 210 first sends a first frame image of a first data stream to be transmitted (4k@30 video stream), then sends a first frame image of each application video stream of a second data stream to be transmitted, then the video capturing device 210 sends a second frame image of the first data stream to be transmitted (4k@30 video stream), and then sends a second frame image of each application video stream of the second data stream to be transmitted, until each frame image of the first data stream to be transmitted and each frame image of the second data stream to be transmitted are transmitted.
In the prior art, after an application video stream with larger resolution is transmitted to a television, other application video streams are transmitted to the television, so that the transmission delay of other application video streams is higher. In other words, with the data stream transmission method provided by the embodiment of the application, the video acquisition device does not need to send the second data stream to be transmitted to the video display device after the transmission of the first data stream to be transmitted is finished, so that the defect that only one video stream can be transmitted independently (after the transmission of one video stream is finished, the other video stream is transmitted) in the prior art is avoided, and the transmission delay of the video stream is reduced.
Therefore, the video capturing apparatus 210 and the video display apparatus 220 may perform the partition transmission or the time-sharing alternating transmission including the channel partition on the plurality of application video streams based on the maximum transmission bandwidth of the second line and the bandwidth information of each channel in the data area, thereby reducing the transmission delay of the video streams and improving the transmission efficiency of the video streams. The partition transmission refers to that a plurality of application video streams are spliced to obtain a data stream to be transmitted, and then the video acquisition device sends the data stream to be transmitted to the video display device, and the content of the partition transmission can be referred to the above description of S240 and S250, which are not repeated here. The above-mentioned time-division alternating transmission including channel partition may be described with reference to fig. 5 and fig. 6, and will not be described herein.
In the above embodiment of the present application, the time-sharing alternate transmission is illustrated by taking the video capturing device to send 2 data streams to be transmitted to 1 video display device as an example, and in some possible embodiments, the 2 data streams to be transmitted may also be sent to different video display devices respectively. For example, the video acquisition device sends a first data stream to be transmitted to a first video display device, and the first video display device plays a first application video stream; the video acquisition device sends the second data stream to be transmitted to the second video display device, and the second video display device plays the application video stream included in the second data stream to be transmitted.
After the video display apparatus 220 receives the data stream to be transmitted, please continue to refer to fig. 2, the data stream transmission method provided in the embodiment of the present application may further include S260.
The video display apparatus 220 processes the data stream to be transmitted and plays at least one application video stream S260.
The step S260 specifically includes: the video display device 220 splits the data stream to be transmitted according to the channel information to obtain all application video streams; further, the video display device 220 selects at least one of all the application video streams to play.
In one possible example, the application video stream played by the video display apparatus 220 is obtained by splitting the data stream to be transmitted by the video display apparatus 220 according to the channel information.
In another possible example, the application video stream played by video display device 220 is: the video display device 220 splits the data stream to be transmitted according to the channel information, and performs AI processing on the split application video stream to obtain an AI data stream, where the AI processing may be, but is not limited to: target detection, face recognition, object classification, color recognition, object counting, etc. For example, the video display device 220 includes a processor that can be used to perform the operations corresponding to the AI process described above and display the results of the operations on the display screen of the video display device 220.
Since the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line and the bandwidth information of each channel, the channel information further includes the number of application video streams in the data stream to be transmitted and the channel and resolution information where each application video stream is located, after the video display device 220 obtains the channel information, computing resources can be allocated for playing the application video stream according to the resolution information of each application video stream, which can improve the utilization rate of the computing resources of the video display device 220 and avoid the occurrence of a clip on the video display device 220. In addition, in the process of splitting the data stream to be transmitted by the video display device 220, the video display device 220 not only can determine the position of the application video stream according to the channel where each application video stream is located, but also can determine the width and height information of the application video stream by using the resolution information of the application video stream included in the channel information, thereby improving the accuracy of the video display device in analyzing the data stream to be transmitted.
Because the data stream to be transmitted comprises a plurality of application video streams, each application video stream can be of different types, after the video display device receives the data stream to be transmitted, the video display device can process and play the data stream to be transmitted, and a user can view various video information according to the original video stream through the video display device, such as a user can view preview stream and photographing stream in video call, and view face recognition information (such as classifying face types of faces) obtained according to the original video stream in the process of video call, so that the viewing experience of the user is improved.
It will be appreciated that, in order to implement the functions of the above embodiments, the video capturing apparatus and the video display apparatus include corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the various illustrative elements and method steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application scenario and design constraints imposed on the solution.
The data stream transmission method provided according to the present embodiment is described in detail above with reference to fig. 1 to 6, and the video capturing apparatus, the video display apparatus, and the terminal device provided according to the present embodiment will be described below with reference to fig. 7 and 8.
Fig. 7 is a schematic diagram of a data stream transmission system according to the present application. The data stream transmission system 700 may be used to implement the functions of the video capturing apparatus 210 and the video display apparatus 220 in the above-described method embodiment, so that the beneficial effects of the above-described method embodiment can be also implemented. In this embodiment, the data streaming system 700 includes a video capturing device 710 and a video display device 720. The video capturing apparatus 710 and the video display apparatus 720 are connected through a first line and a second line, respectively, where the first line is used to transmit a control instruction between the video capturing apparatus 710 and the video display apparatus 720, and the second line is used to transmit a data stream to be transmitted, which is sent by the video capturing apparatus 710 to the video display apparatus 720.
The following description will be made with reference to fig. 7 for the structure and function of the data streaming system 700, where the video capturing apparatus 710 may implement the functions of the video capturing apparatus 210 shown in fig. 2, 5 and 6, and the video display apparatus 720 may implement the functions of the video display apparatus 220 shown in fig. 2, 5 and 6. It should be understood that the present embodiment only exemplarily divides the structure and functional modules of the video capturing apparatus 710 and the video display apparatus 720, and the present application is not limited in any way to the specific division thereof.
As shown in fig. 7, the video capturing apparatus 710 includes a first transceiver unit 711, a first processing unit 712, a splicing unit 713, and a first storage unit 714, which may be used to implement the methods corresponding to the respective operation steps performed by the video capturing apparatus 210 shown in fig. 2, 5, and 6.
When the video capturing apparatus 710 is used to implement the functions of the video capturing apparatus 210 in the method embodiment shown in fig. 2, the first transceiver unit 711 is used to perform S250, the first processing unit 712 is used to perform S230, and the splicing unit 713 is used to perform S240.
When the video capturing apparatus 710 is used to implement the functions of the video capturing apparatus 210 in the method embodiment shown in fig. 5, the first transceiver unit 711 is used to perform S2501, the first processing unit 712 is used to perform S230, and the splicing unit 713 is used to perform S2401 and S2402.
The first storage unit 714 may be configured to store the information such as the plurality of application video streams and the data stream to be transmitted, which are generated in the above steps.
As shown in fig. 7, the video display apparatus 720 includes a second transceiver unit 721, a second processing unit 722, a display unit 723, and a second storage unit 724, which may be used to implement the methods corresponding to the respective operation steps performed by the video display apparatus 220 shown in fig. 2, 5, and 6.
When the video display apparatus 720 is used to implement the functions of the video display apparatus 220 in the method embodiment shown in fig. 2 and 5, the second transceiver unit 721 is used to perform S220, the second processing unit 722 is used to perform S210, and the second processing unit 722 and the display unit 723 are also used to perform S260.
The second storage unit 724 may be configured to store the information such as the acquisition instruction generated in the above step, the plurality of application video streams, and the received data stream to be transmitted.
A more detailed description of the data streaming system 700 may be directly obtained by referring to the related descriptions in the embodiments shown in fig. 2, 5 and 6, and will not be repeated herein.
Fig. 8 is a schematic structural diagram of a terminal device according to the present application, where the terminal device 800 includes a processor 810 and a communication interface 820. Processor 810 and communication interface 820 are coupled to each other. It is understood that communication interface 820 may be a transceiver or an input-output interface. Optionally, the terminal device 800 may further comprise a memory 830 for storing instructions executed by the processor 810 or for storing input data required by the processor 810 to execute instructions or for storing data generated after the processor 810 executes instructions.
When the terminal device 800 is used to implement the methods shown in fig. 2 and 5, the processor 810, the communication interface 820, and the memory 830 may also cooperatively implement various operational steps in a data stream transmission method performed by the video capture apparatus 210 or the video display apparatus 220. The terminal device 800 may also perform the functions of the video capturing apparatus 710 and/or the video display apparatus 720 shown in fig. 7, which are not described herein.
The specific connection medium between the communication interface 820, the processor 810, and the memory 830 is not limited in the embodiment of the present application. In the embodiment of the present application, the communication interface 820, the processor 810 and the memory 830 are connected through the bus 840 in fig. 8, where the bus is indicated by a thick line in fig. 8, and the connection manner between other components is only schematically illustrated, but not limited thereto. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 8, but not only one bus or one type of bus.
The memory 830 may be used for storing software programs and modules, such as program instructions/modules corresponding to the data streaming method provided in the embodiments of the present application, and the processor 810 executes the software programs and modules stored in the memory 830, thereby performing various functional applications and data processing. The communication interface 820 may be used for communication of signaling or data with other devices. The terminal device 800 may have a plurality of communication interfaces 820 in the present application.
In one possible example, the Memory 830 described above may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), and the like.
The processor may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
The method steps in the embodiments of the present application may be implemented by hardware, or may be implemented by executing software instructions by a processor. The software instructions may be comprised of corresponding software modules that may be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. In addition, the ASIC may reside in a terminal device. The processor and the storage medium may reside as discrete components in a terminal device.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user device, or other programmable apparatus. The computer program or instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer program or instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center by wired or wireless means. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The usable medium may be a magnetic medium, e.g., floppy disk, hard disk, tape; optical media, such as digital video discs (digital video disc, DVD); but also semiconductor media such as solid state disks (solid state drive, SSD).
In various embodiments of the application, where no special description or logic conflict exists, terms and/or descriptions between the various embodiments are consistent and may reference each other, and features of the various embodiments may be combined to form new embodiments based on their inherent logic.
In the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural. In the text description of the present application, the character "/", generally indicates that the associated objects are an or relationship; in the formula of the present application, the character "/" indicates that the front and rear associated objects are a "division" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural. at least one (one) of a, b and c may also represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
It will be appreciated that the various numerical numbers referred to in the embodiments of the present application are merely for ease of description and are not intended to limit the scope of the embodiments of the present application. The sequence number of each process does not mean the sequence of the execution sequence, and the execution sequence of each process should be determined according to the function and the internal logic.
Claims (31)
- A data stream transmission method, wherein the method is performed by a video acquisition device, the video acquisition device is connected with a video display device through a first line and a second line, respectively, the first line is used for transmitting control instructions of the video acquisition device and the video display device, and the method comprises:generating a plurality of application video streams according to the original video streams acquired by the video acquisition device;splicing the application video streams to obtain a data stream to be transmitted, wherein the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line;and sending the data stream to be transmitted to the video display device through the second line.
- The method of claim 1, wherein the data stream to be transmitted comprises channel information, the channel information comprising a number of channels in the second line and bandwidth information for each of the channels, the channel information further comprising a number of application video streams in the data stream to be transmitted and a channel in which each of the application video streams is located.
- The method of claim 2, wherein the channel information includes resolution information for each of the application video streams.
- A method according to claim 2 or 3, characterized in that the channel information is arranged in the header of the data stream to be transmitted.
- The method of any of claims 1-4, wherein the plurality of application video streams includes a first application video stream and a second application video stream, the resolution of the first application video stream and the resolution of the second application video stream being different.
- The method of claim 5, wherein the data stream to be transmitted further comprises a data region, the data region comprising a plurality of channels, the concatenating the plurality of application video streams resulting in the data stream to be transmitted comprising:the application video streams are respectively arranged in the corresponding target channels; and the target channel corresponding to the application video stream is a channel with the maximum transmission bandwidth being greater than or equal to the transmission bandwidth required by the application video stream.
- The method of claim 5, wherein the data stream to be transmitted comprises a data region, the data region comprising a plurality of channels, the concatenating the plurality of application video streams resulting in the data stream to be transmitted comprising:If the transmission bandwidth required by the first application video stream is greater than the transmission bandwidth of any one of the channels and is smaller than or equal to the maximum transmission bandwidth of the second line, the first application video stream is used as a first data stream to be transmitted, and the first application video stream is any one of the application video streams;if the transmission bandwidth required by the residual data stream is matched with the transmission bandwidth of the channels, splicing a plurality of application video streams in the residual data stream to obtain a second data stream to be transmitted, wherein the residual data stream comprises other application video streams except the first application video stream in the plurality of application video streams obtained according to the original video stream;the sending the data stream to be transmitted to the video display device through the second line includes:and alternately transmitting the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second line.
- The method of any of claims 1-7, wherein the plurality of application video streams include at least one of a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitor stream, a time-of-flight ranging stream, or a gesture control stream.
- The method of any one of claims 1-8, wherein the first line is a universal serial bus, USB, and the second line is a high definition multimedia interface, HDMI, line.
- A data stream transmission method, wherein the method is performed by a video display device, the video display device is connected to a video acquisition device through a first line and a second line, respectively, the first line is used for transmitting control instructions of the video acquisition device and the video display device, and the method comprises:receiving a data stream to be transmitted sent by the video acquisition device through the second line, wherein the data stream to be transmitted is obtained by splicing a plurality of application video streams, and the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line;and processing the data stream to be transmitted and playing at least one application video stream.
- The method of claim 10, wherein the data stream to be transmitted comprises channel information, the channel information comprising a number of channels in the second line and bandwidth information for each of the channels, the channel information further comprising a number of application video streams in the data stream to be transmitted and a channel in which each of the application video streams is located.
- The method of claim 11, wherein the channel information includes resolution information for each of the application video streams.
- The method according to claim 11 or 12, characterized in that the channel information is provided in the header of the data stream to be transmitted.
- The method according to any of claims 11-13, wherein said processing said data stream to be transmitted and playing at least one of said application video streams comprises:splitting the data stream to be transmitted according to the channel information to obtain all application video streams;and selecting at least one application video stream in all the application video streams for playing.
- The utility model provides a video acquisition device, its characterized in that, video acquisition device is connected with video display device through first circuit and second circuit respectively, first circuit is used for transmitting video acquisition device with video display device's control command, video acquisition device includes:the processing unit is used for generating a plurality of application video streams according to the original video streams acquired by the video acquisition device;the splicing unit is used for splicing the application video streams to obtain a data stream to be transmitted, and the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line;And the receiving and transmitting unit is used for transmitting the data stream to be transmitted to the video display device through the second line.
- The apparatus of claim 15, wherein the data stream to be transmitted comprises channel information, the channel information comprising a number of channels in the second line and bandwidth information for each of the channels, the channel information further comprising a number of application video streams in the data stream to be transmitted and a channel in which each of the application video streams is located.
- The apparatus of claim 16, wherein the channel information comprises resolution information for each of the application video streams.
- The apparatus according to claim 16 or 17, wherein the channel information is provided at a header of the data stream to be transmitted.
- The apparatus of any of claims 15-18, wherein the plurality of application video streams includes a first application video stream and a second application video stream, the first application video stream having a different resolution than the second application video stream.
- The apparatus according to claim 19, wherein the data stream to be transmitted further comprises a data area, the data area comprises a plurality of channels, and the splicing unit is specifically configured to set the plurality of application video streams in respective corresponding target channels; and the target channel corresponding to the application video stream is a channel with the maximum transmission bandwidth being greater than or equal to the transmission bandwidth required by the application video stream.
- The apparatus of claim 19, wherein the data stream to be transmitted includes a data area, the data area includes a plurality of channels, and the splicing unit is specifically configured to take a first application video stream as a first data stream to be transmitted if a transmission bandwidth required by the first application video stream is greater than a transmission bandwidth of any one of the plurality of channels and is less than or equal to a maximum transmission bandwidth of the second line, and the first application video stream is any one of the plurality of application video streams;the splicing unit is specifically configured to splice a plurality of application video streams in a residual data stream to obtain a second data stream to be transmitted if a transmission bandwidth required by the residual data stream is matched with a transmission bandwidth of the plurality of channels, where the residual data stream includes other application video streams except the first application video stream in the plurality of application video streams obtained according to the original video stream;the transceiving unit is specifically configured to alternately send the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second line.
- The apparatus of any of claims 15-21, wherein the plurality of application video streams include at least one of a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitor stream, a time-of-flight ranging stream, or a gesture control stream.
- The apparatus of any one of claims 15-22, wherein the first line is a universal serial bus, USB, and the second line is a high definition multimedia interface, HDMI, line.
- The utility model provides a video display device, its characterized in that, video display device is connected with video acquisition device through first circuit and second circuit respectively, first circuit is used for transmitting video acquisition device with video display device's control command, video display device includes:the receiving and transmitting unit is used for receiving a data stream to be transmitted sent by the video acquisition device through the second line, the data stream to be transmitted is obtained by splicing a plurality of application video streams, and the transmission bandwidth required by the data stream to be transmitted is smaller than or equal to the maximum transmission bandwidth of the second line;the processing unit is used for processing the data stream to be transmitted to obtain a plurality of application video streams;And the display unit is used for playing at least one application video stream in the plurality of application video streams.
- The apparatus of claim 24, wherein the data stream to be transmitted comprises channel information, the channel information comprising a number of channels in the second line and bandwidth information for each of the channels, the channel information further comprising a number of application video streams in the data stream to be transmitted and a channel in which each of the application video streams is located.
- The apparatus of claim 25, wherein the channel information comprises resolution information for each of the application video streams.
- The apparatus according to claim 25 or 26, wherein the channel information is provided at a header of the data stream to be transmitted.
- The apparatus according to any one of claims 25-27, wherein the processing unit is specifically configured to split the data stream to be transmitted according to the channel information to obtain all application video streams;the display unit is specifically configured to select at least one application video stream from the all application video streams for playing.
- A terminal device comprising a processor and interface circuitry, the processor receiving or transmitting data through the interface circuitry, the processor for implementing the method of any one of claims 1 to 9, or the method of any one of claims 10 to 14, through logic circuitry or execution of code instructions.
- A computer readable storage medium, characterized in that the storage medium has stored therein a computer program which, when executed by a terminal device or a processor, implements the method of any of claims 1 to 9 or the method of any of claims 10 to 14.
- A computer program product comprising instructions which, when run on a terminal device or processor, implement the method of any of claims 1 to 9 or the method of any of claims 10 to 14.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2021/083112 WO2022198605A1 (en) | 2021-03-25 | 2021-03-25 | Data stream transmission method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116711298A true CN116711298A (en) | 2023-09-05 |
Family
ID=83396263
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202180088320.5A Pending CN116711298A (en) | 2021-03-25 | 2021-03-25 | Data stream transmission method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116711298A (en) |
| WO (1) | WO2022198605A1 (en) |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110209183A1 (en) * | 2010-02-25 | 2011-08-25 | Samsung Electronics Co., Ltd. | Method and apparatus for transmitting data based on audio/video interface |
| CN202663483U (en) * | 2012-05-11 | 2013-01-09 | 青岛海尔电子有限公司 | Television set and control system of television set |
| CN104683863A (en) * | 2013-11-28 | 2015-06-03 | ***通信集团公司 | Method and equipment for multimedia data transmission |
| CN106911343B (en) * | 2017-02-21 | 2019-01-29 | 广州土圭垚信息科技有限公司 | A kind of VR device and VR system |
| CN108924520B (en) * | 2018-09-29 | 2024-04-02 | 深圳市道通智能航空技术股份有限公司 | Transmission control method, device, controller, shooting equipment and aircraft |
| CN109889752A (en) * | 2019-02-01 | 2019-06-14 | 深圳市炬力北方微电子有限公司 | Transmission device and throwing screen display methods |
-
2021
- 2021-03-25 CN CN202180088320.5A patent/CN116711298A/en active Pending
- 2021-03-25 WO PCT/CN2021/083112 patent/WO2022198605A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022198605A1 (en) | 2022-09-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11190715B2 (en) | System and method for event camera data processing | |
| WO2022042211A1 (en) | Service processing method and device | |
| US10212339B2 (en) | Image generation method based on dual camera module and dual camera apparatus | |
| KR102495763B1 (en) | Electronic device and method for correcting images corrected by a first image processing mode in external electronic device using a second image processing mode | |
| WO2019001091A1 (en) | Method and device for video transmission, and computer readable storage medium | |
| KR20200011000A (en) | Device and method for augmented reality preview and positional tracking | |
| KR20200092631A (en) | Apparatus and method for generating slow motion video | |
| CN115526787A (en) | Video processing method and device | |
| US20220301278A1 (en) | Image processing method and apparatus, storage medium, and electronic device | |
| US10769416B2 (en) | Image processing method, electronic device and storage medium | |
| US10552980B2 (en) | Image processing apparatus, image processing method, and storage medium | |
| US20150181167A1 (en) | Electronic device and method for video conference management | |
| US9807255B2 (en) | Image processing apparatus | |
| CN113962873A (en) | Image denoising method, storage medium and terminal device | |
| CN116711298A (en) | Data stream transmission method and device | |
| TW202114398A (en) | Image transmission device, image display system with remote screen capture function, and remote screen capture method | |
| WO2021213017A1 (en) | Video superimposition method, apparatus and system | |
| CN115967784A (en) | Image transmission processing system and method based on MIPI CSI-PHY protocol | |
| CN109274955B (en) | Compression and synchronization method and system for light field video and depth map, and electronic equipment | |
| KR20200091196A (en) | Method for augmented reality contents data optimization using user feedback | |
| CN117156293B (en) | Photographing method and related device | |
| WO2021208053A1 (en) | Image processing method and apparatus, camera module, and movable device | |
| WO2024082863A1 (en) | Image processing method and electronic device | |
| CN116028383B (en) | Cache management method and electronic equipment | |
| WO2024066521A1 (en) | Memory refresh method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |