WO2021056575A1 - 一种低延迟信源信道联合编码方法及相关设备 - Google Patents
一种低延迟信源信道联合编码方法及相关设备 Download PDFInfo
- Publication number
- WO2021056575A1 WO2021056575A1 PCT/CN2019/109220 CN2019109220W WO2021056575A1 WO 2021056575 A1 WO2021056575 A1 WO 2021056575A1 CN 2019109220 W CN2019109220 W CN 2019109220W WO 2021056575 A1 WO2021056575 A1 WO 2021056575A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- code stream
- reference frame
- image
- frame image
- channel
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
Definitions
- This application relates to the field of wireless transmission technology, and in particular to a low-delay source-channel joint coding method and related equipment.
- wireless networks With the development of wireless transmission technology, it is possible to use wireless networks to carry high-definition video, but at the same time, wireless networks carry high-definition video also put forward new requirements for image coding and decoding technology, because of the short-time and time-varying characteristics of wireless channels during wireless transmission. It will cause rapid changes in channel capacity, so the image codec algorithm needs to be able to quickly track and adapt to this change, so that the delay and image quality at the receiving end can be maintained at an acceptable level.
- JPEG Joint Photographic Experts Group
- MPEG Moving Picture Experts Group
- the embodiments of the present application provide a low-latency source-channel joint coding method and related equipment. By optimizing image coding or transmission mode, information loss can be reduced during video transmission, so as to avoid the receiving end from being unable to play clear and complete videos. Improve user viewing experience.
- the embodiments of the present application provide a low-delay source-channel joint coding method, which may include: down-sampling the target image to obtain a reference frame image and a non-reference frame image; Images are encoded separately to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image; based on the channel environment of the current wireless channel, the first channel resource and the second channel resource are determined, The first channel resource is used to send the first code stream and the second channel resource is used to send the second code stream, respectively, wherein the first channel resource is better than the second channel resource.
- Video wireless transmission technology mainly includes several main links such as video encoding, wireless transmission, video decoding and display.
- the encoding end can down-sample the target image to obtain the reference frame image and the non-reference frame image, and then perform image encoding on the reference frame image to obtain the first bit stream.
- the first channel resource and the second channel resource are determined according to the channel environment of the current wireless channel, and the first channel resource is better than the second channel resource, that is, it can be understood that the reference frame image can be encoded during wireless transmission.
- the second code stream is sent hierarchically. For example, you can set the priority of the first code stream to be higher than the priority of the second code stream, and then send the first code stream separately With the second code stream, according to the channel environment of the current wireless channel, the service quality of the channel when the first code stream is sent is higher than the service quality of the channel when the second code stream is sent.
- the encoder can send the more important first code stream first, and then send the second code stream with a lower level; or, when the current wireless channel has a poor channel environment, it can send After receiving the first code stream or after a period of time after sending the first code stream, the end sends the second code stream with a lower level. Based on the channel environment of the current wireless channel, the encoding end sends the first code stream and the second code stream separately, so that when the receiving end cannot receive the complete code stream, the probability that the receiving end receives the first code stream can be improved. The first code stream to get a complete reconstructed image.
- the target image is down-sampled, and after obtaining multiple copies of lower resolution images, the image encoding is performed, which can make the code stream obtained after down-sampling smaller than the direct code stream.
- the size of the code stream obtained when the target image is image-encoded makes it easier for the decoder at the receiving end to receive the complete code stream. Therefore, after the encoding end performs image encoding on the target image through downsampling, the encoded stream information is based on the channel environment of the current wireless channel and sent through the wireless channel respectively, so that the encoded stream information can still adapt to real-time changes.
- the channel environment further reduces the probability of information loss during wireless transmission, thereby avoiding the inability of the receiving end to reconstruct a complete video image due to information loss, and improving the user’s viewing experience.
- the above-mentioned image encoding is performed on the reference frame image and the non-reference frame image respectively, to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoding
- the latter second code stream includes: performing intra-frame compression on each of the reference frame images included in the reference frame image to obtain the first code stream; performing the frame image with the residual difference between the non-reference frame image and the reference frame image Inner compression obtains the corresponding above-mentioned second code stream.
- the encoding end can select a simpler down-sampling to layer the target image and reduce the resolution.
- the intra-frame compression encoding method used when encoding the layered duplicate image because the frame Inner compression is spatial compression.
- the embodiment of the present application does not use inter-frame compression when performing image encoding on the down-sampled replica image.
- the use of intra-frame compression can cause the channel capacity to jitter and drop below the initial bit rate, so that the decoder at the receiving end can receive the bit stream information encoded based on the smaller-resolution copy image image, and can be based on The reconstructed image obtained from the code stream information.
- the channel capacity is reduced, the probability of information loss in the transmission process is reduced, and the phenomenon of mosaic, stuttering, and blurring of the video image information is avoided, and the user's perception and experience are improved.
- the above-mentioned channel environment also includes channel capacity; the above-mentioned image coding is performed on the above-mentioned reference frame image and the above-mentioned non-reference frame image to obtain the first code stream after the coding of the above-mentioned reference frame image, and the above-mentioned non-reference frame image.
- the second code stream after the reference frame image it further includes: determining the coding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the coding parameters are used to control the corresponding image in the image processing.
- a corresponding code stream is generated according to the target code rate, and the target code rates respectively corresponding to the first code stream and the second code stream are less than or equal to the channel capacity.
- the encoding end can obtain the channel capacity of the current channel through the transmitter when the channel capacity changes, and adjust the encoding parameters of the encoder when encoding the duplicate image according to the feedback channel capacity information. , So that the code rate of the coded code stream can be adapted to the current channel capacity, and the target code stream can be sent smoothly, reducing the loss of the code stream due to the reduced channel capacity.
- the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; the foregoing is based on current To determine the first channel resource and the second channel resource, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, including: based on the current channel environment , Determine the transmission parameters corresponding to the first code stream and/or the second code stream, wherein the transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the first code stream
- the sending parameter of is better than the sending parameter of the second code stream; the first code stream is sent according to the sending parameter corresponding to the first code stream, and the second code stream is sent according to the sending parameter corresponding to the second code stream.
- the encoder can obtain the channel environment of the current wireless channel through the transmitter when the channel changes, such as: channel bandwidth, signal to interference plus noise ratio, signal to noise ratio, received signal strength indicator , Duty cycle, bit rate, etc.), according to the channel environment to adjust the transmission parameters (such as: transmission power, modulation and coding strategy information, etc.) when sending the first code stream and/or when sending the second code stream, it is understandable Yes, after adjusting the sending parameters according to the channel information, the sending parameters of the first code stream are different from the sending parameters of the second code stream, and the sending parameters of the first code stream can make the transmitter more expensive than the sending parameters of the second code stream.
- the transmission parameters such as: transmission power, modulation and coding strategy information, etc.
- the first code stream is sent at a faster transmission speed and higher quality of service (QoS). Therefore, in the process of wireless transmission, the channel resource of the first code stream is better than the channel resource of the second code stream to be sent, that is, the transmission quality when sending the first code stream is higher than when sending the second code stream. Transmission quality, or, preferentially send more important code streams (that is, code streams that have a small number of duplicate images after image encoding), causing jitter in channel capacity, etc., causing the decoder at the receiving end to fail to receive the complete code stream. Because of the characteristics of intra-frame compression, it can avoid the inability of the receiver to reconstruct the complete video image due to the loss of information. The decoder is only based on the reconstructed image obtained by the partial code stream received first, which can reduce the mosaic and stutter of the video information. , Blur and other phenomena to enhance the user’s perception and experience.
- QoS quality of service
- the wireless channel includes a first wireless channel and a second wireless channel; the first channel resource and the second channel resource are determined based on the current channel environment, and the first channel resource is used to transmit the
- the first code stream and the sending of the second code stream using the second channel resource include: determining the first wireless channel and the second wireless channel based on the current channel environment, and passing the first code stream through the first wireless channel Sending, sending the above-mentioned second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the above-mentioned first wireless channel is higher than that of the above-mentioned second wireless channel.
- the encoding end may classify the video source during the wireless transmission process, and the channel resource of the first code stream is better than the channel resource of the second code stream for transmission.
- the first code stream is mapped to a video (Video, VI) service
- the second code stream is mapped to a best effort (Best Effort, BE) service.
- the first code stream and the second code stream sent through the wireless channel use different QoS guarantee levels, which increases the probability of receiving the first code stream at the receiving end, so that the decoder at the receiving end can
- the image can still be reconstructed only through the first code stream to avoid the inability of the receiving end to reconstruct the complete video image due to information loss and reduce the video information Mosaic, stuttering, blurring, etc. appear to enhance the user’s perception and experience.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned reference The code stream of the frame image; the audio information in the target video is obtained; each frame image of the multi-frame image included in the target video corresponds to the code stream of the reference frame image and the audio information through the first wireless channel.
- the encoding end can obtain all target images in the target video included in the multi-frame target image.
- the code streams corresponding to the above-mentioned reference frame image and audio information are mapped to the VI service transmission with a high quality of service guarantee mechanism.
- the above audio information may be audio information that has been encoded by voice.
- the decoder at the receiving end can reconstruct the video based only on the code streams and audio information corresponding to all target images received through the channel with a high quality of service guarantee mechanism, so as to increase the probability of receiving the first code stream at the receiving end. Avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, reduce the phenomenon of no sound and freeze in the video information, and improve the user's visual experience.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned non- The code stream of the reference frame image; the code stream of each frame image in the multi-frame image included in the target video corresponding to the non-reference frame image is sent through the second wireless channel.
- the encoding end may separately map to the BE service transmission with a slightly lower service quality assurance mechanism after acquiring the code streams of all target images corresponding to the aforementioned non-reference frame images in the multi-frame target images included in the target video.
- the decoder can only be based on the results of all target images received through the channel with a high quality of service guarantee mechanism.
- the code stream and audio information corresponding to the first copy reconstruct the video to prevent the receiving end from being unable to reconstruct the complete video image due to the loss of information, reduce the phenomenon of mosaic, stutter, and blur in the video information, and improve the user's perception and experience.
- the embodiments of the present application provide another low-delay source-channel joint coding method, which may include: down-sampling the target image to obtain reference frame images and non-reference frame images; according to the current wireless channel channel environment, Determine the coding parameters corresponding to the reference frame image and the non-reference frame image; according to the coding parameters, perform image encoding on the reference frame image and the non-reference frame image respectively to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoding After the second code stream; send the first code stream and the second code stream separately.
- the encoding end may first downsample the target image to obtain the reference frame image and the non-reference frame image; then according to the channel environment of the current wireless channel, determine the encoding parameters corresponding to the reference frame image and the non-reference frame image, Then use the coding parameters to encode the reference frame image and the non-reference frame image, so that the bit rates of the first bit stream and the second bit stream are lower than the channel capacity of the current wireless channel; finally, the first bit stream and the second bit stream are The code streams are sent out through the wireless channel respectively.
- adjusting the encoding parameters during image encoding can make the code rate of the generated code stream lower than the channel capacity of the current wireless channel, and the code stream will not be due to the low channel capacity when the code stream is wirelessly transmitted.
- the complete bit stream cannot be transmitted, or the channel capacity is jittered, or the channel capacity is reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate)
- the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate
- the image coding and decoding algorithm of the embodiment of the present application can quickly track and adapt to changes in the channel environment of the wireless channel, so that the delay and image quality of the receiving end can be maintained at an acceptable level.
- the encoding end first down-samples the target image, and then performs image encoding after obtaining multiple copies of lower resolution images, which can make the code stream required after the down-sampling is obtained. It is smaller than the size of the code stream obtained when directly encoding the target image, which makes it easier for the decoder at the receiving end to receive the complete code stream.
- the code stream information obtained by encoding the image according to the channel information of the current wireless channel can be adapted to the real-time changing channel environment, thereby reducing the probability of information loss during transmission and preventing the receiving end from being unable to reconstruct complete information due to information loss.
- the aforementioned encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate when the corresponding image is encoded; the aforementioned encoding parameters are used to compare the aforementioned reference frame image and the aforementioned non-reference frame image Image encoding is performed separately to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded, including: each of the reference frame images included in the reference frame image according to the target The code rate is compressed within the frame to obtain the first code stream; the residuals of each of the non-reference frame images contained in the non-reference frame image and all the reference frame images included in the reference frame image are in accordance with the target bit rate Perform the above-mentioned intra-frame compression to obtain the corresponding above-mentioned second code stream.
- the encoding end can select a simpler down-sampling to layer the target image and reduce the resolution.
- the intra-frame compression encoding method used when encoding the layered duplicate image because the frame Inner compression is spatial compression.
- the encoding end uses intra-frame compression on the down-sampled copy image, which can cause the channel capacity to jitter and reduce to below the initial bit rate, so that the decoder at the receiving end can accept the copy based on a smaller resolution.
- the code stream information of the image image can be reconstructed from the code stream information.
- the channel capacity is reduced, the probability of information loss in the transmission process is reduced, and the phenomenon of mosaic, stuttering, and blurring of the video image information is avoided, and the user's perception and experience are improved.
- the above-mentioned channel environment includes one or more of the bandwidth of the channel, the signal to interference plus noise ratio, the signal-to-noise ratio, the received signal strength indicator, the duty cycle, and the bit rate;
- the method further includes: determining the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the transmission parameters are used to follow the target For modulation and coding strategy information and/or target transmission power to transmit a code stream, the transmission parameters of the first code stream are better than the transmission parameters of the second code stream.
- the encoder can obtain the channel environment of the current wireless channel through the transmitter when the channel changes, and adjust the transmission when sending the first code stream and/or the second code stream according to the channel environment.
- Parameters such as transmission power, modulation and coding strategy information, etc.
- the transmission parameters of the first code stream are different from the transmission parameters of the second code stream.
- the transmission parameters enable the transmitter to transmit the first code stream at a faster transmission speed and a higher quality of service (QoS).
- the encoding end sends the first code stream using channel resources better than the channel resources of the second code stream, that is, the transmission quality when sending the first code stream is higher than when sending the second code stream.
- the transmission quality at the time, or the more important code stream is sent first that is, the code stream after the image is encoded with a small number of duplicate images), so that the decoder at the receiving end can still receive the complete code stream when the channel capacity is jittered.
- the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned sending the above-mentioned first code stream and the above-mentioned second code stream respectively includes: passing the above-mentioned first code stream through a first wireless channel.
- Channel transmission is to send the above-mentioned second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the above-mentioned first wireless channel is higher than that of the above-mentioned second wireless channel.
- the encoding end may classify the video source during the wireless transmission process, and the channel resource of the first code stream is better than the channel resource of the second code stream for transmission.
- the first code stream and the second code stream sent by the encoding end through the wireless channel use different QoS guarantee levels, which increases the probability that the receiving end receives the first code stream and makes the receiving end decode
- the receiver can still reconstruct the image only through the first bit stream, avoiding the inability of the receiving end to reconstruct the complete video image due to loss of information, which reduces Mosaic, stuttering, blurring and other phenomena appear in the video information, which improves the user's perception and experience.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned reference The code stream of the frame image; the audio information in the target video is obtained; each frame image of the multi-frame image included in the target video corresponds to the code stream of the reference frame image and the audio information through the first wireless channel.
- the encoding end can obtain all target images in the target video included in the multi-frame target image.
- the code streams corresponding to the above-mentioned reference frame image and audio information are mapped to the VI service transmission with a high quality of service guarantee mechanism.
- the aforementioned audio information may be audio information encoded by voice.
- the decoder at the receiving end can reconstruct the video based only on the code streams and audio information corresponding to all target images received through the channel with a high quality of service guarantee mechanism, so as to increase the probability of receiving the first code stream at the receiving end. Avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, reduce the phenomenon of no sound and freeze in the video information, and improve the user's visual experience.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned non-target image.
- the code stream of the reference frame image; the code stream of each frame image in the multi-frame image included in the target video corresponding to the non-reference frame image is sent through the second wireless channel.
- the encoding end may separately map to the BE service transmission with a slightly lower service quality assurance mechanism after acquiring the code streams of all target images corresponding to the aforementioned non-reference frame images in the multi-frame target images included in the target video. Therefore, even if the code streams corresponding to the non-reference frame images of all target images lose part or even all of the code stream information when the channel changes, the decoder can only be based on the results of all target images received through the channel with a high quality of service guarantee mechanism.
- the code stream and audio information corresponding to the first copy reconstruct the video to prevent the receiving end from being unable to reconstruct the complete video image due to the loss of information, reduce the phenomenon of mosaic, stutter, and blur in the video information, and improve the user's perception and experience.
- embodiments of the present application provide a low-delay source-channel joint decoding method, which may include: receiving a first code stream sent by an encoding end, where the first code stream is a code stream obtained after image encoding of a reference frame image , Wherein the reference frame image includes one or more images obtained after down-sampling the target image; after decoding the first code stream, the reference frame image corresponding to the first code stream is obtained; according to the reference frame image, reconstruct Target image.
- the decoding end ie, the receiving end
- the decoding end can receive the first code stream sent by the encoding end, and then decode the first code stream to obtain the first code stream.
- a reference frame image corresponding to a code stream is then reconstructed based on the reference frame image.
- the first code stream is the code stream obtained after image encoding of the reference frame image, where the reference frame image includes one or more images obtained after down-sampling the target image. Therefore, the code received by the receiving end The stream is obtained by down-sampling the target image and then image encoding.
- downsampling can sample the target image into multiple copies of lower resolution, part of them can be used as the reference frame image.
- the code stream size obtained by down-sampling and re-encoding the target image is smaller than the code stream size obtained when the target image is directly encoded. Therefore, the code stream after down-sampling the target image may better adapt to changes in the wireless channel. It also reduces the probability of information loss during the transmission process, and at the same time avoids the inability of the receiving end to reconstruct the complete video image due to the loss of information.
- the target image is any one of multiple target images included in the target video;
- the reconstruction of the target image according to the reference frame image includes: according to the reference frame image and the target image At least one of the reference frame image and the non-reference frame image corresponding to an adjacent frame of target image is reconstructed by using an interpolation algorithm.
- the receiving end can use the interpolation algorithm to map the reference frame image corresponding to the first code stream to the target image of the previous frame of the target image corresponding to the first code stream when the receiving end only receives the first code stream.
- the reference frame image is reconstructed into the target image.
- the receiving end when the receiving end receives the first bit stream, it can avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, and reduce the appearance of no sound and freeze in the video information. Phenomenon to enhance the user’s perception and experience.
- the method further includes: within a preset time period after receiving the first code stream, receiving a second code stream sent by the encoding end, wherein the second code stream is non-
- the reference frame image is the code stream obtained after the image encoding, and the non-reference frame image includes the remaining images except the reference frame image obtained after down-sampling the target image; if the second code stream is incomplete, the first After the two bit streams are decoded, the corresponding incomplete non-reference frame image is obtained; according to the incomplete non-reference frame image, the peripheral pixels of the incomplete non-reference frame image are determined; and the target image is reconstructed according to the reference frame image, including : According to the surrounding pixels of the reference frame image and the incomplete non-reference frame image, the target image is reconstructed by an interpolation algorithm.
- the receiving end can determine whether to receive the second code stream within the preset time period of receiving the first code stream, if not, according to the first code stream and the adjacent frame of target image Corresponding to at least one of the reference frame image and the non-reference frame image, reconstruct the target image; if the second code stream is received, determine whether the received second code stream is incomplete, if the received second code stream is incomplete
- the incomplete second code stream and the complete first code stream can be used to interpolate to obtain the target image, which avoids the inability of the receiving end to reconstruct the complete video image due to partial information loss, and reduces the phenomenon of no sound and freezing in the video information. Improve the user's perception experience.
- the method further includes: if it is determined that the second code stream is complete, after image decoding the second code stream, the corresponding non-reference frame image is obtained; , Reconstructing the target image includes: splicing the reference frame image and the non-reference frame image to reconstruct the target image.
- an embodiment of the present application provides a low-delay source-channel joint coding device, which is characterized by comprising: an encoder and a transmitter, wherein the encoder is used for down-sampling the target image to obtain a reference Frame images and non-reference frame images; image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoded The second code stream; a transmitter, used to determine a first channel resource and a second channel resource based on the channel environment of the current wireless channel, and respectively use the first channel resource to send the first code stream and use the first channel resource The second code stream is sent with two channel resources, wherein the first channel resource is better than the second channel resource.
- the encoder is configured to perform image encoding on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image
- the encoder is specifically configured to: perform intra-frame compression on each of the reference frame images included in the reference frame image to obtain the first code stream; combine the non-reference frame image with the reference frame The residual of the image is subjected to the intra-frame compression to obtain the corresponding second code stream.
- the above-mentioned channel environment also includes channel capacity; the above-mentioned transmitter performs image encoding on the above-mentioned reference frame image and the above-mentioned non-reference frame image respectively to obtain the first code after encoding the above-mentioned reference frame image Before the second code stream after the encoding of the non-reference frame image, it is also used to determine the encoding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the encoding parameters are used to control When the corresponding image is encoded, the corresponding code stream is generated according to the target code rate, and the target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.
- the above-mentioned channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate;
- the above-mentioned transmitter When used to determine the first channel resource and the second channel resource based on the current channel environment, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, respectively, It is specifically used for: determining the transmission parameters corresponding to the first code stream and/or the second code stream respectively based on the current channel environment, wherein the transmission parameters are used for target modulation and coding strategy information and/or target transmission Power transmission code stream, the transmission parameters of the first code stream are better than the transmission parameters of the second code stream; the first code stream is transmitted according to the transmission parameters corresponding to the first code stream, and the transmission parameters corresponding to the second code stream are Sending parameters to send the above second code stream.
- the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned transmitter is used in the above-mentioned current channel environment to determine the first channel resource and the second channel resource, and respectively use
- the first channel resource is used to send the first code stream
- the second channel resource is used to send the second code stream, it is specifically used to: determine the first wireless channel and the second wireless channel based on the current channel environment, and The first code stream is sent through a first wireless channel, and the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned encoder is further used for: acquiring the corresponding target image of each frame of the multi-frame target image included in the above-mentioned target video The code stream of the reference frame image; the audio information in the target video is obtained; the transmitter is also used to: pass each frame image of the multi-frame image included in the target video corresponding to the code stream of the reference frame image and the audio information The above-mentioned first wireless channel transmission.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned encoder is further used for: acquiring the corresponding target image of each frame of the multi-frame target image included in the above-mentioned target video The code stream of the non-reference frame image; the transmitter is further used to: transmit the code stream of each frame image corresponding to the non-reference frame image in the multi-frame image included in the target video through the second wireless channel.
- an embodiment of the present application provides another low-delay source-channel joint coding device, which is characterized by comprising: an encoder and a transmitter, wherein the encoder is used to down-sample the target image to obtain Reference frame image and non-reference frame image; according to the channel environment of the current wireless channel, determine the coding parameters corresponding to the reference frame image and non-reference frame image; according to the coding parameters, perform image coding on the reference frame image and the non-reference frame image to obtain The first code stream after the coding of the reference frame image and the second code stream after the coding of the non-reference frame image; the transmitter is used to send the first code stream and the second code stream respectively.
- the aforementioned encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate during image encoding; the encoder is used to compare the aforementioned reference frame image and the aforementioned reference frame image according to the aforementioned encoding parameters.
- the non-reference frame image When the non-reference frame image is encoded separately to obtain the first code stream after the reference frame image is encoded, and the second code stream after the non-reference frame image is encoded, it is specifically used to:
- Each of the reference frame images is intra-compressed according to the target code rate to obtain the first code stream; the residual difference between the non-reference frame image and the reference frame image is subjected to the intra-frame compression according to the target code rate to obtain the corresponding The above second code stream.
- the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate;
- the foregoing transmitter Used to separately transmit the first code stream and the second code stream, and also used to: determine the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the The sending parameter is used to send the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the sending parameter of the first code stream is better than the sending parameter of the second code stream.
- the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned transmitter is used to transmit the above-mentioned first code stream and the above-mentioned second code stream, and is specifically used to: The first code stream is sent through a first wireless channel, and the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned encoder is further used for: acquiring the corresponding target image of each frame of the multi-frame target image included in the above-mentioned target video The code stream of the reference frame image; the audio information in the target video is obtained; the transmitter is also used to: pass each frame image of the multi-frame image included in the target video corresponding to the code stream of the reference frame image and the audio information The above-mentioned first wireless channel transmission.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned encoder is further used for: acquiring the corresponding target image of each frame of the multi-frame target image included in the above-mentioned target video The code stream of the non-reference frame image; the transmitter is further used to: transmit the code stream of each frame image corresponding to the non-reference frame image in the multi-frame image included in the target video through the second wireless channel.
- an embodiment of the present application provides a low-delay source-channel joint coding device, which is characterized by comprising: a first sampling unit for down-sampling a target image to obtain a reference frame image and a non-reference frame image ;
- the first coding unit is used to perform image coding on the reference frame image and the non-reference frame image respectively, to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded;
- the first transmission The unit is used to determine the first channel resource and the second channel resource based on the channel environment of the current wireless channel, and respectively use the first channel resource to send the first code stream and the second channel resource to send the second code stream, where the first The channel resource is better than the second channel resource.
- the above-mentioned first encoding unit is specifically configured to perform intra-frame compression on each of the above-mentioned reference frame images included in the above-mentioned reference frame image to obtain the above-mentioned first code stream;
- the residuals of each of the non-reference frame images and all the reference frame images included in the reference frame image are subjected to the intra-frame compression to obtain the corresponding second code stream.
- the foregoing channel environment further includes channel capacity;
- the device further includes: a first encoding parameter unit, configured to perform image encoding on the foregoing reference frame image and the foregoing non-reference frame image separately to obtain Before the first code stream after the encoding of the reference frame image and the second code stream after the encoding of the non-reference frame image, based on the channel capacity, the encoding parameters corresponding to the reference frame image and the non-reference frame image are determined, wherein, The encoding parameter is used to control the corresponding image to generate a corresponding code stream according to the target code rate when the image is encoded.
- the target code rate corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.
- the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate;
- the unit is specifically configured to determine the transmission parameters corresponding to the first code stream and/or the second code stream respectively based on the current channel environment, wherein the transmission parameters are used to modulate and encode strategy information according to the target and/or the target.
- the code stream is transmitted at the transmit power, and the transmission parameters of the first code stream are better than the transmission parameters of the second code stream; the first code stream is transmitted according to the transmission parameters corresponding to the first code stream, and the second code stream corresponds to The sending parameters of sending the above second code stream.
- the wireless channel includes a first wireless channel and a second wireless channel; the first sending unit is specifically configured to: determine the first wireless channel and the second wireless channel based on the current channel environment , Sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a first acquisition unit configured to acquire each of the multi-frame target images included in the above-mentioned target video.
- One frame of target image corresponds to the code stream of the above-mentioned reference frame image; acquiring the audio information in the above-mentioned target video; the second sending unit is used to correspond each frame of the multi-frame image included in the above-mentioned target video to the code stream of the above-mentioned reference frame image
- the above-mentioned audio information is transmitted through the above-mentioned first wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a second acquiring unit configured to acquire each of the multi-frame target images included in the above-mentioned target video.
- One frame of the target image corresponds to the code stream of the non-reference frame image; the third sending unit is used to pass the code stream of each frame image corresponding to the non-reference frame image in the multi-frame image included in the target video through the second wireless channel send.
- an embodiment of the present application provides another low-delay source-channel joint coding device, which is characterized in that it includes: a second sampling unit for down-sampling the target image to obtain a reference frame image and a non-reference frame Image; the second coding parameter unit is used to determine the coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; the second coding unit is used to compare the reference frame image and the non-reference frame image according to the coding parameters Frame images are encoded separately to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image; the fourth sending unit is used to send the first code stream and the second code stream respectively .
- the foregoing encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate during image encoding; the foregoing second encoding unit is specifically used to: Each of the aforementioned reference frame images is intra-compressed according to the aforementioned target code rate to obtain the aforementioned first code stream; the residual difference between the aforementioned non-reference frame image and the aforementioned reference frame image is subjected to the aforementioned intra-frame compression according to the aforementioned target code rate to obtain the corresponding The above second code stream.
- the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate;
- the fourth The sending unit is configured to separately send the first code stream and the second code stream, and is also used to: determine the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein The foregoing transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the transmission parameters of the first code stream are better than the transmission parameters of the second code stream.
- the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned fourth sending unit is specifically configured to: send the above-mentioned first code stream through the first wireless channel, and transmit the above-mentioned second code stream through the first wireless channel.
- the code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a third acquisition unit configured to acquire each of the multi-frame target images included in the above-mentioned target video.
- One frame of target image corresponds to the code stream of the aforementioned reference frame image; acquiring the audio information in the aforementioned target video; and the fifth sending unit is used to correspond each frame of the multi-frame image included in the aforementioned target video to the code stream of the aforementioned reference frame image
- the above-mentioned audio information is transmitted through the above-mentioned first wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a fourth acquisition unit configured to acquire each of the multi-frame target images included in the above-mentioned target video.
- One frame of target image corresponds to the code stream of the aforementioned non-reference frame image;
- the sixth sending unit is configured to pass the code stream of each frame image corresponding to the aforementioned non-reference frame image in the multi-frame image included in the aforementioned target video through the second wireless channel send.
- an embodiment of the present application provides a low-delay source-channel joint decoding device, which is characterized by comprising: a receiving unit, configured to receive a first code stream sent by an encoding end, and the first code stream is a reference frame image The code stream obtained after image encoding, wherein the reference frame image includes one or more images obtained after down-sampling the target image; the decoding unit is used to decode the first code stream to obtain the first code stream Corresponding to the reference frame image; the image unit is used to reconstruct the target image according to the reference frame image.
- the above-mentioned target image is any one of multiple frames of target images included in the target video; the above-mentioned image unit is specifically used for: according to a frame of target image adjacent to the above-mentioned target image according to the above-mentioned reference frame image Corresponding to at least one of the reference frame image and the non-reference frame image, the above-mentioned target image is reconstructed through an interpolation algorithm.
- the device further includes: a fifth acquiring unit, configured to receive the second code stream sent by the encoding end within a preset time period after receiving the first code stream, where
- the second code stream is a code stream obtained by encoding a non-reference frame image, and the non-reference frame image includes images obtained after down-sampling the target image except for the reference frame image; if the first The second code stream is incomplete, and the second code stream is decoded to obtain the corresponding incomplete non-reference frame image; the peripheral pixels of the incomplete non-reference frame image are determined according to the incomplete non-reference frame image; the image unit, It is specifically used for: reconstructing the target image through an interpolation algorithm based on the surrounding pixels of the reference frame image and the incomplete non-reference frame image.
- the device further includes: a sixth acquiring unit, configured to, if it is determined that the second code stream is complete, perform image decoding on the second code stream to obtain the corresponding non-reference frame image
- the image unit is specifically used for: splicing the reference frame image and the non-reference frame image to reconstruct the target image.
- an embodiment of the present application provides a service device, the service device includes a processor, and the processor is configured to support the service device to execute the low-latency source-channel joint coding method provided in the first aspect or the foregoing
- the second aspect provides a corresponding function in another low-delay source-channel joint coding method.
- the service device may further include a memory, which is used for coupling with the processor, and stores the necessary program instructions and data of the service device.
- the service device may also include a communication interface for the service device to communicate with other devices or a communication network.
- an embodiment of the present application provides a service device, the service device includes a processor, and the processor is configured to support the service device to execute the corresponding low-latency source-channel joint decoding method provided in the third aspect. Function.
- the service device may further include a memory, which is used for coupling with the processor, and stores the necessary program instructions and data of the service device.
- the service device may also include a communication interface for the service device to communicate with other devices or a communication network.
- an embodiment of the present application provides a computer program, which includes instructions, when the computer program is executed by a computer, the computer can execute the low-latency source-channel joint coding provided in the fourth aspect.
- the embodiments of the present application provide a computer program, the computer program includes instructions, when the computer program is executed by a computer, the computer can execute the low-latency source channel joint decoding provided in the sixth aspect. The process performed by the device.
- an embodiment of the present application provides a computer storage medium for storing the low-latency source-channel joint coding device provided in the fourth aspect, or another low-latency signal provided in the fifth aspect.
- the source-channel joint coding device, or the computer software instructions used by the low-latency source-channel joint decoding device provided in the sixth aspect described above, includes a program for executing the program designed in the foregoing aspect.
- an embodiment of the present application provides a chip system, which includes a processor, and is configured to support a service device to implement the functions involved in the first, second, or third aspects described above.
- the chip system further includes a memory, and the memory is used to store program instructions and data necessary for the data sending device.
- the chip system can be composed of chips, or include chips and other discrete devices.
- an embodiment of the present application provides an electronic device, including the processing chip provided by any one of the foregoing first aspect, second aspect, or third aspect, and a discrete device coupled to the chip.
- Fig. 1 is a schematic structural diagram of a low-delay source-channel joint coding system provided by an embodiment of the present application.
- Fig. 2A is a schematic flowchart of a low-delay source-channel joint coding method provided by an embodiment of the present application.
- Fig. 2B is a schematic diagram of an application scenario provided by the present application.
- FIG. 2C is a schematic diagram of a first code stream and a second code stream respectively sent through the first channel and the second channel provided by an embodiment of the present application.
- FIG. 3A is a schematic flowchart of another low-delay source-channel joint coding and decoding method provided by an embodiment of the present application.
- FIG. 3B is a schematic diagram of an application for encoding and decoding a target image provided by an embodiment of the present application.
- Fig. 3C is a schematic diagram of another application scenario provided by the present application.
- Fig. 4A is a schematic structural diagram of a low-delay source-channel joint coding apparatus provided by an embodiment of the present application.
- Fig. 4B is a schematic structural diagram of another low-delay source-channel joint coding apparatus provided by an embodiment of the present application.
- Fig. 4C is a schematic structural diagram of a low-delay source-channel joint decoding apparatus provided by an embodiment of the present application.
- Fig. 5 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
- component used in this specification are used to denote computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution.
- the component may be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and/or a computer running on a processor.
- the application running on the computing device and the computing device can be components.
- One or more components may reside in processes and/or threads of execution, and components may be located on one computer and/or distributed among two or more computers.
- these components can be executed from various computer readable media having various data structures stored thereon.
- the component may be based on, for example, data having one or more data packets (for example, data from two components interacting with another component in a local system, a distributed system, and/or a network, for example, the Internet that interacts with other systems through signals) Signals are communicated through local and/or remote processes.
- data packets for example, data from two components interacting with another component in a local system, a distributed system, and/or a network, for example, the Internet that interacts with other systems through signals
- Signals are communicated through local and/or remote processes.
- JPEG Joint Photographic Experts Group
- ISO International Standardization Organization
- IEC International Electrotechnical Commission
- MPEG Moving Picture Experts Group
- MPEG-1 MPEG-1
- MPEG-2 MPEG-2
- MPEG-4 MPEG-4
- MPEG-7 MPEG-21
- the MPEG standard video compression coding technology mainly uses the inter-frame compression coding technology with motion compensation to reduce the time redundancy, the discrete cosine transform technology to reduce the spatial redundancy of the image, and the entropy coding is used in the aspect of information representation. Reduced statistical redundancy.
- RGB image R stands for red; G stands for green; B stands for blue.
- R stands for red
- G stands for green
- B stands for blue.
- red, green, and blue Each pixel of the color image is composed of red and green.
- the different proportions of, blue, such an image is an RGB image.
- Wireless Local Area Network which is a very convenient data transmission system. It uses radio frequency (RF) technology and uses electromagnetic waves to replace the old obstructive twisted-pair copper wire (Coaxial).
- RF radio frequency
- Coaxial copper wire
- the formed local area network is connected in the air, so that the wireless local area network can use a simple access structure to allow users to use it to achieve the ideal state of "information portable and convenient to travel the world".
- Discrete Cosine Transform is a transformation related to Fourier Transform, similar to Discrete Fourier Transform, but using only real numbers.
- the discrete cosine transform is equivalent to a discrete Fourier transform whose length is about twice as long. This discrete Fourier transform is performed on a real even function (because the Fourier transform of a real even function is still a real even function ), in some deformations, it is necessary to move the input or output position by half a unit.
- Run Length Coding also known as “Run Length Coding” or “Run Length Coding”
- RLC Run Length Coding
- the basic principle of run-length coding is to replace consecutive symbols with the same value with a symbol value or string length (continuous symbols form a continuous "run”. Run-length coding is named after it), so that the length of the symbol is less than that of the original data. length. Only when the code of each row or each column of data changes, the code and the number of repetitions of the same code are recorded at a time, so as to achieve data compression.
- Signal to Interference plus Noise Ratio refers to: Signal to Interference plus Noise Ratio (SINR) is the strength of the received useful signal and the received interference signal (noise and interference) The ratio of the intensity.
- SNR Signal to Noise Ratio
- the signal here refers to the electronic signal from the outside of the device that needs to be processed by this device, and the noise refers to the irregular extra signal (or information) that does not exist in the original signal generated after passing through the device. The signal does not change with the change of the original signal.
- Packet Error Ratio is an indicator that measures the accuracy of data packets (data packets) transmission within a specified time.
- Macroblock is a basic concept in video coding technology.
- a coded image is usually divided into several macroblocks, and a macroblock is composed of a luminance pixel block and two additional chrominance pixel blocks.
- the luminance block is a 16x16 pixel block
- the size of the two chrominance image pixel blocks depends on the image sampling format. For example, for a YUV420 sampled image, the chrominance block is an 8x8 pixel block.
- several macroblocks are arranged in the form of slices, and the video coding algorithm uses macroblocks as the unit to encode macroblocks one by one and organize them into a continuous video stream. .
- Data rate refers to the data flow used by a video file in a unit time, also called bit rate, which is the most important part of picture quality control in video encoding.
- Entropy coding is a lossless data compression scheme independent of the specific characteristics of the medium.
- One of the main types of entropy coding is to create and assign a unique prefix code to each input symbol, and then replace each fixed-length input symbol with the corresponding variable-length prefix-free (prefix-free) The output code word is replaced, so as to achieve the purpose of compressing data.
- the length of each codeword is approximately proportional to the negative logarithm of the probability. Therefore, the shortest code is used for the most common symbols.
- Intraframe compression also known as spatial compression. When compressing a frame of image, only the data of the current frame is considered without considering the redundant information between adjacent frames, which is actually similar to still image compression. Intra-frame compression generally uses a lossy compression algorithm.
- Interframe compression is based on the fact that the two consecutive frames before and after many videos or animations have great correlation, or the characteristics of little change in the information of the two frames before and after. That is, continuous video has redundant information between adjacent frames. According to this feature, compressing the redundancy between adjacent frames can further increase the amount of compression and reduce the compression ratio. Inter-frame compression is also called temporal compression (temporal compression).
- MCS Modulation and Coding Scheme
- Down-sampling a sample sequence is sampled once at intervals of several samples, so that the new sequence obtained is the down-sampling of the original sequence.
- the sampling rate changes are mainly due to different modules of signal processing may have different sampling rate requirements.
- down-sampling still has to satisfy the sampling theorem, otherwise such down-sampling will cause aliasing of signal components.
- Down-sampling is decimation, which is one of the basic contents in multi-rate signal processing.
- Image coding is also called image compression. It refers to a technology that uses a small number of bits to represent an image or the information contained in an image under the condition that a certain quality (signal-to-noise ratio requirement or subjective evaluation score) is met.
- QoS Quality of Service
- YUV is a color coding method. Often used in various video processing components. When YUV encodes photos or videos, it takes into account human perception and allows the bandwidth of chroma to be reduced. YUV is a type of compiling true-color color space (color space). Proper nouns such as Y'UV, YUV, YCbCr, YPbPr, etc. can all be called YUV, which overlap with each other. "Y” means brightness (Luminance, Luma), "U” and “V” mean chrominance, density (Chrominance, Chroma)
- Channel capacity also known as channel capacity, refers to the minimum upper bound of the achievable rate when information can be reliably transmitted in a channel.
- the so-called reliable transmission refers to the transmission of information with an arbitrarily small error rate.
- the channel capacity is the limit information rate of a given channel that can be reached with an arbitrarily small error probability.
- the unit of channel capacity is bit per second, knight per second, etc.
- wireless video transmission solves the problem of data transmission such as audio and video in the case of inconvenient wiring construction. Therefore, wireless projection compresses the screen information of terminals such as mobile phones/tablets, and then sends them to large-screen devices such as TVs wirelessly, shares videos/pictures with friends through wireless networks, and conducts video conferences through wireless networks.
- image coding and decoding through wireless transmission.
- a common wireless video transmission system generally consists of two parts: the transmitter and the receiver.
- an appropriate gain antenna can be added to increase the transmission distance according to actual needs.
- the transmitter and receiver of wireless video transmission mainly include audio and video encoding, wireless transmission, audio and video decoding and display.
- JPEG standard coding and MPEG standard coding are two common image coding methods. Both of these image coding methods can be converted into YUV space after the original image undergoes color space transformation, and then undergoes segmentation and discrete cosine transformation. After (Discrete Cosine Transform, DCT), quantization, and entropy coding, a compressed image is obtained.
- DCT Discrete Cosine Transform
- quantization quantization
- entropy coding a compressed image is obtained.
- the audio and video codec algorithm at the transmitting end needs to be able to quickly track and adapt to this change, so that the delay and image quality of the receiving end, that is, the receiving end, can be kept at an acceptable level.
- traditional audio and video coding technologies such as JPEG standard coding and MPEG standard coding, which only focus on compression ratio and image quality, cannot quickly make corresponding coding adjustments according to channel capacity. Therefore, it is easy to lose part or all of the code stream information during wireless transmission.
- wireless transmission generally does not care about the priority of the source part (that is, the code stream information after the audio and video is encoded), and the transmission is performed without difference.
- the encoding end cannot control whether the key information (such as audio information, stream information of important frames) that may be lost in the wireless channel, and cannot quickly adapt to changes in the wireless channel, and has no control over the information source (such as : Stream information or other data information) is sent hierarchically, and the receiving end may not be able to reconstruct a complete video image due to the loss of important information.
- key information such as audio information, stream information of important frames
- information source such as : Stream information or other data information
- FIG. 1 is a schematic structural diagram of a low-delay source-channel joint coding system provided by an embodiment of the present application.
- the architecture of the low-delay source-channel joint coding system in this application may include the coding end 10 and the decoding end 11 in FIG. 1. among them.
- the encoding end 10 of the low-latency source-channel joint encoding system architecture includes a video source 101, an encoder 102, and a transmitter 103.
- the decoding end 11 of the low-latency source-channel joint coding system architecture includes a decoder 111 and a display device 112. among them,
- the video source 101 is an interface, display memory, storage, etc. used to provide target video.
- the target video can come from various types of interfaces, such as: High Definition Multimedia Interface (HDMI) interface, digital video (Display Port, DP) ) Interface, video graphics array standard (Video Graphics Array, VGA) interface, etc.
- HDMI High Definition Multimedia Interface
- DP digital video
- VGA Video Graphics Array
- the target video may be sent to the encoder 102, so that the encoder 102 encodes the target image of the target video.
- the encoder 102 is a device that compiles and converts a signal (such as a bit stream) or data into a signal form that can be used for communication, transmission, and storage.
- the encoder can convert angular displacement or linear displacement into electrical signals.
- the former is called a code disc and the latter is called a code ruler.
- the encoder can be related coding software running on a general-purpose central processing unit (CPU) or a dedicated chip On or on an independent encoding chip, or a part of an independent chip, such as a part of a System-on-a-Chip (SoC) chip of an integrated circuit.
- CPU general-purpose central processing unit
- SoC System-on-a-Chip
- the target image can be hierarchically encoded, that is, the target image is down-sampled to obtain the reference frame image and the non-reference frame image for image encoding; and then the first image after the reference frame image encoding is obtained.
- the code stream, and the second code stream after encoding the non-reference frame image; finally, the first code stream and the second code stream are sent to the decoder 111 through the transmitter 103.
- the coding parameters may also receive coding parameters fed back according to the channel information of the current wireless channel sent through the coding parameter feedback control, so as to make the bit rate of the code stream after image coding smaller than the channel capacity of the current wireless channel.
- the target image can also be down-sampled to obtain the reference frame image and the non-reference frame image; according to the current wireless channel channel environment, determine the corresponding encoding of the reference frame image and the non-reference frame image Parameters; according to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image.
- the transmitter 103 is a device that uses radio waves as a data transmission device using a wirelessly connected local area network, and the transmission distance is generally tens of meters.
- the transmitter 103 may be a WiFi chip, and may include a driver program, firmware, etc. that are matched with the WiFi chip.
- the encoder 102 can perform image encoding to obtain the first code stream and then the second code stream is sent through the wireless channel respectively. Among them, the channel resources when the first code stream is sent wirelessly are high. Channel resources when sending the second code stream.
- the transmitter 103 can also obtain the channel environment of the current wireless channel, such as: channel capacity, channel bandwidth, or signal to interference plus noise ratio, etc.; and then send channel information to the encoder 103 through coding parameter feedback control, so that The encoder 103 adjusts the encoding parameters of the first code stream and the second code stream in real time; the transmitter 103 can also adjust the sending parameters of the first code stream and the second code stream in real time.
- the encoding parameter feedback control can be used as an algorithm inside or outside the encoder, which may be implemented by software or chip.
- the encoding parameter feedback control is mainly used to adjust the image of the encoder according to the channel information of the current wireless channel. Encoding parameters during encoding. It is understandable that the encoder 102 and the transmitter 103 may be devices coupled together to receive target video information sent from the video source 101, and send the target video information to the decoder 111 after image encoding.
- a decoder 111 is a hardware/software device that can decode and restore digital video and audio data streams into analog video and audio signals.
- the encoder mainly compresses analog video and audio signals into data-encoded files, while the decoder converts the data-encoded files into analog video and audio signals.
- the decoder can be related decoding software running on a general-purpose CPU or a dedicated chip, or an independent decoding chip, or a part of an independent chip (such as a SoC chip); for example: when the decoder 111 is an independent decoding chip, the decoder 111 can receive and receive The first code stream sent by the transmitter 103 of the encoding end, the first code stream is the code stream obtained after image encoding of the reference frame image, wherein the reference frame image includes the reference frame image obtained after down-sampling the target image; then decode The device 111 decodes the first code stream to obtain a reference frame according to the reverse flow of the process when the encoder 102 encodes the target image of the target video; finally obtains the target image according to the reference frame.
- the decoder 111 when the decoder 111 is an independent decoding chip, the decoder 111 can receive and receive The first code stream sent by the transmitter 103 of the encoding end, the first code stream is the code stream obtained after image encoding of
- the decoder 111 may also receive the second code stream, and then decode the second code stream according to the reverse process when the encoder 102 encodes the target image of the target video to obtain a complete or incomplete non-reference frame; finally, the decoder uses The reference frame and the complete non-reference frame are spliced into a complete reconstructed frame, that is, the target image; when the non-reference frame is incomplete, adjacent frame images (such as the reference frame image and the non-reference frame image of an adjacent target image) can be used At least one of) or a set of surrounding pixels is interpolated to obtain a reconstructed frame, and then the obtained reconstructed frame is sent to the display device 112 for display and audio playback.
- adjacent frame images such as the reference frame image and the non-reference frame image of an adjacent target image
- the decoder 111 determines whether the non-reference frame is incomplete, it can also determine whether the code stream is incomplete when receiving the code stream, and then determine whether the image corresponding to the code stream is incomplete, which is not limited in this application. .
- the display device 112 is a display tool that displays certain data on a screen through a specific transmission device and then reflects it to the human eye.
- it can refer to a display, a projector, a virtual reality head-mounted display device, and a belt.
- Devices such as smart terminals with display functions.
- the display device 112 may be: a virtual reality head-mounted display device (such as virtual reality (VR) glasses, VR goggles, VR helmets), smart phones, notebook computers, tablet devices, projectors, cameras, etc., Or display clients, applications, etc. installed or running on computers, tablet devices, smart phones.
- VR virtual reality
- the display device when the display device is a projector, the projector may project the target video data information sent by the decoder 111 onto the screen through the projection device, and then play the target video.
- the decoder 111 and the display device 112 may be devices coupled together to receive the code stream information sent from the transmitter 103, decode the code stream information and display the target video corresponding to the code stream information.
- decoding end 11 in the low-delay source-channel joint coding system architecture provided by the embodiment of the present application is equivalent to the receiving end in the present application.
- the low-latency source-channel joint coding system architecture of FIG. 1 is only a partial exemplary implementation in the embodiments of the present application.
- the low-latency source-channel joint coding system architecture in the embodiments of the present application includes but not only Limited to the above low-latency source-channel joint coding system architecture.
- this application proposes a low-latency source-channel joint coding method, which can avoid the requirement for stream integrity in traditional audio and video coding and decoding algorithms, and also It can quickly track changes in channel capacity, keeping the receiving end delay and image quality at an acceptable level.
- the channel capacity is reduced, the phenomenon of mosaic, stuttering, and blurring in the video information is reduced, and the user's visual experience is improved.
- FIG. 2A is a schematic flowchart of a low-latency source-channel joint coding method according to an embodiment of the present application, which can be applied to the low-latency source-channel joint coding system architecture described in FIG. 1 Among them, the encoder 102 and transmitter 103 can be used to support and execute steps S201 to S203 of the method flow shown in FIG. 2A.
- Step S201 down-sampling the target image to obtain a reference frame image and a non-reference frame image.
- the encoder 102 down-samples the target image to obtain the reference frame image and the non-reference frame image.
- the down-sampling is to sample a sequence of samples at intervals of several samples, so that the new sequence obtained is the down-sampling of the original sequence.
- FIG. 2B is a schematic diagram of an application scenario provided by this application.
- the encoder 102 can down-sample the input target video with multiple frames of target images, where the down-sampling horizontal sampling rate is 2:1, and the vertical sampling rate is 2:1.
- the ratio is 2:1, and it is divided into 4 lower-resolution duplicate images.
- one duplicate image can be selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images.
- the reference frame image may include one or more images
- the non-reference frame image may also include one or more images
- the number of images included in the reference frame image is less than or equal to the number of images included in the non-reference frame image.
- the code rate can then be jittered or reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the code rate of the preset size) , So that the decoder at the receiving end can still reconstruct the image based on the smaller-resolution copy image.
- the target video includes multiple frames of target images, and the target image is any one of the multiple frames of target images.
- the encoder 102 down-samples the target image to obtain N duplicate images, the resolution of the N duplicate images is lower than the resolution of the target image, where N is a positive integer greater than 1; Divided into reference frame image and non-reference frame image.
- the multiple copies of lower resolution images can be divided into two types of images with different numbers.
- the number of images included in the reference frame image is less than or equal to the number of images included in the non-reference frame image.
- the intersection of the reference frame image and the non-reference frame image is an empty set.
- the four duplicate images obtained after down-sampling, among them, any one duplicate image can be selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images.
- Step S202 Perform image encoding on the reference frame image and the non-reference frame image, respectively, to obtain a first code stream after the reference frame image is coded, and a second code stream after the non-reference frame image is coded.
- the encoder 102 performs image encoding on the reference frame image and the non-reference frame image after the multiple copy images are divided, to obtain the first code stream after the reference frame image is encoded, and the second code after the non-reference frame image is encoded. flow.
- the encoder 102 may sequentially perform block, DCT, quantization, run-length coding, entropy coding, packing, marking, etc., on the copy image in the reference frame image to generate the first code stream.
- the encoder 102 can perform blocking, DCT, quantization, and navigation on the residual of each duplicate image in the non-reference frame image relative to the duplicate image in the reference frame image or directly on each duplicate image in the non-reference frame image. Long coding, entropy coding, packing, marking, etc. generate the second code stream.
- the encoder 102 may perform intra-frame compression on the reference frame image and the non-reference frame image, respectively, to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded. It is understandable that while choosing a simpler down-sampling method to layer the target image and reduce the resolution, the intra-frame compression encoding method used when encoding the layered duplicate image, because intra-frame compression is spatial compression. When compressing a frame of image, only the data of the current frame is considered without considering the redundant information between adjacent frames, and the inter-frame compression should refer to other frame data. The compression rate is large.
- this application implements For example, when the down-sampled copy image is coded, the inter-frame compression is not applicable. Therefore, using the intra-frame compression method can cause the decoder to receive the code stream information based on the smaller resolution copy image when the channel capacity is jittered and drops below the initial bit rate, and can be based on the code The reconstructed image obtained from the stream information. At the same time, when the channel capacity is reduced, the phenomenon of mosaic, stuttering, and blurring in the video information is reduced, and the user's perception and experience are improved.
- the encoder 102 may also use other layered encoding methods to divide the target image into lower-resolution duplicate images, and the resolution, bit rate, and frame rate among the multiple duplicate images may be different. , Can also be the same.
- the code stream generated by layered video coding contains multiple sub-streams.
- the sub-streams are divided into a basic layer and an extended layer. Each layer has a different bit rate, frame rate and resolution.
- the basic layer has the most basic video quality.
- Each subsequent extension layer is a supplement to the previous one.
- the streaming end can be based on the actual network environment (such as: channel bandwidth, signal and interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle Ratio, bit rate, etc.) to select several sub-streams for decoding.
- the encoder 102 performs image encoding on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoding the first bit stream.
- the coding parameters corresponding to the reference frame image and the non-reference frame image may also be determined based on the channel environment of the current wireless channel, where the channel environment may include channel capacity, and the coding parameters are used
- the target code rate corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.
- the encoding parameter determined by the encoder may be an encoding parameter sent by the encoding parameter feedback control. Therefore, when the channel capacity changes, the channel capacity of the current channel can be obtained through the transmitter, and the encoding parameters of the encoder when encoding the duplicate image can be adjusted according to the channel capacity information fed back, so that the encoded bit stream
- the code rate can be adapted to the current channel capacity, and the target code stream can be sent smoothly, reducing the loss of the code stream due to the decrease of the channel capacity.
- Step S203 Determine the first channel resource and the second channel resource based on the channel environment of the current wireless channel, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, respectively.
- the transmitter 103 may use the first code stream and the second code stream received from the encoder 102 after determining the first channel resource and the second channel resource based on the channel environment of the current wireless channel.
- the channel resource is used to send the first code stream
- the second channel resource is used to send the second code stream.
- the embodiment of the present application may separately send the first code stream and the second code stream based on the channel environment of the current wireless channel that is fed back.
- the quality of service of the channel when streaming is higher than the quality of service of the channel when transmitting the second code stream.
- the second code stream with a lower level is sent.
- the transmitter 103 may send link statistical information such as SINR/SNR/RSSI to the encoding end via the air interface, and the encoding end makes statistics on the error sub-frame rate, collision rate, etc., and then based on the link statistical information and the error sub-frame rate , Collision rate, etc. Adjust sending parameters, such as sending MCS, transmitting power, quantization parameters, etc.
- link statistical information such as SINR/SNR/RSSI
- the encoding end makes statistics on the error sub-frame rate, collision rate, etc., and then based on the link statistical information and the error sub-frame rate , Collision rate, etc.
- sending parameters such as sending MCS, transmitting power, quantization parameters, etc.
- the channel information of the current channel (such as: channel bandwidth, signal-to-interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, bit rate, etc.) is obtained through the transmitter, According to the channel information, the transmission parameters (such as transmission power, modulation and coding strategy information, etc.) when transmitting the first code stream and/or the second code stream are adjusted.
- the transmitter 103 may be based on the current channel environment , Determine the transmission parameters corresponding to the first code stream and/or the second code stream, wherein the transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, the The transmission parameters of the first code stream are better than the transmission parameters of the second code stream; the first code stream is transmitted according to the transmission parameters corresponding to the first code stream, and the transmission parameters corresponding to the second code stream are transmitted Sending the second code stream.
- the channel environment includes one or more of channel bandwidth, signal-to-interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate.
- the transmitter 103 when the transmitter 103 sends the first code stream and the second code stream, it will respond to the sending request of the first code stream more quickly.
- the importance and priority of the video source are classified, that is, the transmission quality when the first code stream is sent is higher than the transmission quality when the second code stream is sent, or the more important code stream is sent first ( That is, the number of duplicate images is small and the code stream after image encoding), so that when the channel capacity is jittered, and the decoder cannot receive the complete code stream, the reconstructed image can be obtained only based on the duplicate image corresponding to the partial code stream.
- the transmitter 103 may also determine the first wireless channel and the second wireless channel based on the current channel environment, send the first code stream through the first wireless channel, and send the second code stream The stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the wireless channel includes a first wireless channel and a second wireless channel.
- FIG. 2C is a schematic diagram of a first code stream and a second code stream being sent through a first channel and a second channel, respectively, according to an embodiment of the present application.
- the transmitter 103 transmits the first code stream through the first wireless channel, and transmits the second code stream through the second wireless channel.
- the first code stream and the second code stream sent through the wireless channel use different QoS guarantee levels, and the decoder can only receive the first code stream sent through the channel with a high quality of service guarantee mechanism. , And then can reconstruct the image only through the first bit stream, reduce the phenomenon of mosaic, freeze, and blur in the video information, and improve the user's perception and experience. It is understandable that after adjusting the transmission parameters according to the channel information, the transmission parameters of the first code stream are different from the transmission parameters of the second code stream, and the transmission parameters of the first code stream can be compared with the transmission parameters of the second code stream. The machine sends the first stream with a faster sending speed and higher QoS.
- the first code stream is sent with better channel resources and a higher transmission success rate.
- the success rate may refer to the probability that the code stream information is completely transmitted to the receiving end. Therefore, in the process of wireless transmission, the video source can be classified, and the more important stream can be sent first, so that when the channel capacity is jittered, or the decoder cannot receive the complete stream, it can only be based on partial stream correspondence.
- the reconstructed image obtained from the copy image of the image for example: the first code stream and the second code stream are sent over the wireless channel, and the first code stream is mapped to a higher quality of service (QoS) guarantee mechanism, such as:
- QoS quality of service
- the first code stream is mapped to the VI service
- the second code stream is mapped to the BE service.
- the first code stream and the second code stream can also be sent through the same wireless channel, but only the first code stream can be sent first to ensure that the receiving end can receive at least the first code stream.
- the first code stream can be sent first, and only after the first code stream is received at the receiving end or after a preset period of time after the first code stream is sent, The second code stream is sent to ensure that the receiving end can obtain at least one complete, lower-resolution code stream information in order to reconstruct the image.
- Sending the first code stream and the second code stream separately through the wireless channel can increase the probability of the receiving end receiving the first code stream even when the receiving end cannot receive the complete code stream, so that the receiving end can only receive the first code stream. You can also get a complete reconstructed image when the first code stream is used.
- the code stream of each target image corresponding to the reference frame image in the multiple frames of target images included in the target video obtain the audio information in the target video and mark it as audio information;
- Each of the included multi-frame images corresponds to the code stream of the reference frame image and the audio information is sent through the first wireless channel.
- the audio information is as important as the first code stream information of the multi-frame target image. Therefore, in order to avoid the loss of audio information, the first code stream can be sent together with the audio information through a QoS guarantee level comparison. High first channel transmission, reducing the phenomenon of no sound and freezing in the video information, and improving the user's visual experience.
- the code stream of the non-reference frame image is sent through the second wireless channel.
- the decoder can only be based on the results of all target images received through the channel with a high quality of service guarantee mechanism.
- the code stream and audio information corresponding to the first copy reconstruct the video, reduce the phenomenon of mosaic, stutter, and blur in the video information, and improve the user's perception and experience.
- the target image can be down-sampled to obtain the reference frame image and the non-reference frame image, and then the reference frame image is image-encoded to obtain the first code stream, and the non-reference frame image is image-encoded to obtain the second code stream.
- Code stream finally using the first channel resource to send the first code stream, and using the second channel resource to send the second code stream.
- the first channel resource and the second channel resource are determined according to the channel environment of the current wireless channel, and the first channel resource is better than the second channel resource, that is, it can be understood that the reference frame image can be encoded during wireless transmission.
- the second code stream is sent hierarchically.
- the service quality of the channel when the first code stream is sent is higher than the service quality of the channel when the second code stream is sent.
- the second code stream with a lower level is sent.
- the first code stream and the second code stream are sent separately, so that when the receiving end cannot receive the complete code stream, the probability that the receiving end receives the first code stream can be increased, and the received code stream can be used.
- the first code stream obtains a complete reconstructed image.
- the target image is down-sampled, and after obtaining multiple copies of lower resolution images, the image encoding is performed, which can make the code stream obtained after down-sampling smaller than the direct code stream.
- the size of the code stream obtained when the target image is image-encoded makes it easier for the decoder at the receiving end to receive the complete code stream.
- the encoded stream information is based on the current wireless channel channel environment and sent through the wireless channel respectively, so that the encoded stream information can still adapt to the real-time changing channel environment , Thereby reducing the probability of information loss during wireless transmission, thereby preventing the receiving end from being unable to reconstruct a complete video image due to information loss, and improving the user's viewing experience.
- FIG. 3A is a schematic flowchart of a low-latency source-channel joint coding and decoding method provided by an embodiment of the present application
- FIG. 3B is a schematic diagram of a target image encoding and decoding application provided by an embodiment of the present application
- the method shown in FIG. 3A can be applied to the low-delay source-channel joint coding system architecture described in FIG. 1, where the encoder 102 and the transmitter 103 can be used to support and execute the method flow steps shown in FIG. 3A S301-step S304, the decoder 111 can be used to support and execute the method flow step S305-step S307 shown in FIG. 3A.
- the method may include the following steps S301-S307.
- Step S301 down-sampling the target image to obtain a reference frame image and a non-reference frame image.
- step S301 may correspond to the related description of step S201 in FIG. 2A, which will not be repeated here.
- Step S302 Determine the coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel.
- the encoder 102 determines the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel.
- FIG. 3C is another application scenario provided by this application.
- the schematic diagram shows that the coding parameters for graphics coding can be determined according to the channel environment of the current wireless channel. For example, when a 3-second target video needs to be wirelessly transmitted, the encoder 102 can down-sample the input target video with multiple frames of target images, where the down-sampling horizontal sampling rate is 2:1, and the vertical sampling rate is 2:1. The ratio is 2:1, and it is divided into 4 lower-resolution duplicate images.
- one duplicate image is selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images, and then based on the channel environment of the current wireless channel, The encoding parameters corresponding to the reference frame image and the non-reference frame image during image encoding are determined, so as to reduce the size of the code stream after encoding.
- the transmitter 103 may obtain the channel environment of the current wireless channel, where the channel environment includes the channel capacity.
- the channel capacity of the current channel is obtained through the transmitter, and the coding parameters of the encoder when encoding the duplicate image can be adjusted according to the channel capacity, so that the code rate of the coded stream is adapted to Under the current channel capacity, the target code stream can be sent smoothly, reducing the loss of the code stream due to the decrease of the channel capacity.
- Step S303 Perform image encoding on the reference frame image and the non-reference frame image respectively according to the encoding parameters to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image.
- the encoder 102 adjusts the encoding parameters corresponding to the reference frame image and the non-reference frame image during image encoding according to the channel capacity through the encoding parameter feedback control, and then, according to the encoding parameter determined after adjustment, the reference frame image Image coding is performed separately with the non-reference frame image to obtain a first code stream after the reference frame image is coded, and a second code stream after the non-reference frame image is coded.
- the encoding parameters are used to control the corresponding copy to generate the corresponding code stream according to the target code rate during image encoding, so that the code rates of the first code stream and the second code stream are less than or equal to the above-mentioned channel capacity to ensure the code after encoding
- the bit rate of the stream is adapted to the current channel capacity. Therefore, according to the channel information of the current wireless channel, adjusting the coding parameters during image encoding can make the code rate of the generated code stream lower than the channel capacity of the current wireless channel, and the code stream will not be due to the low channel capacity when the code stream is wirelessly transmitted.
- the complete bit stream cannot be transmitted, or the channel capacity is jittered, or the channel capacity is reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate)
- the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate
- the image coding and decoding algorithm of the embodiment of the present application can quickly track and adapt to changes in the channel environment of the wireless channel, so that the delay and image quality of the receiving end can be maintained at an acceptable level.
- the code rates of the first code stream and the second code stream may be the same or different.
- the code rates of the first code stream and the second code stream are the same, the coding parameters of the image coding can be uniformly adjusted.
- the code rate of the code stream is different from that of the second code stream, it is first required to ensure that the code rate of the first code stream is less than or equal to the channel capacity, so as to reduce the probability of code stream loss due to reduced channel capacity.
- Step S304 Send the first code stream and the second code stream respectively.
- step S304 may correspond to the related description of step S203 in FIG. 2A, which will not be repeated here.
- Step S305 Receive the first code stream sent by the encoding end.
- the decoder 111 receives a first code stream sent by the encoding terminal 102, where the first code stream is a code stream obtained after image encoding of a reference frame image.
- the embodiment of the present application may receive the first code stream sent by the encoding end during the wireless transmission, and then decode the first code stream to obtain the reference frame image corresponding to the first code stream, and then according to the reference frame image, Obtain the target image.
- the first code stream is a code stream obtained after image encoding of a reference frame image, where the reference frame image includes one or more images obtained after down-sampling the target image. Therefore, the code received by the encoding end The stream is obtained by down-sampling the target image and then image encoding.
- Downsampling can divide the target image into multiple copies of lower resolution, and select a part of them as the reference frame image. Moreover, the size of the code stream obtained by down-sampling and re-encoding the target image is smaller than the code stream size obtained when the target image is directly coded, and the code stream after down-sampling may be better adapted to changes in the wireless channel, and is transmitted In the process, the probability of information loss is reduced, and the receiving end cannot reconstruct the complete video image due to the loss of information.
- Step S306 After image decoding is performed on the first code stream, a reference frame image corresponding to the first code stream is obtained.
- the decoder 111 obtains the reference frame image corresponding to the first code stream after image decoding of the first code stream.
- the encoder 102 performs block, DCT, quantization, run-length encoding, entropy encoding, packing, and marking of the reference frame image to obtain the corresponding first code stream; then the decoder 111 can follow the process completely opposite to that of the encoder 102, namely , Unmarking, unpacking, anti-entropy coding, inverse run length coding, inverse quantization, inverse DCT, decode the first bit stream to get YUV, and then convert YUV to RGB to get the reference frame image.
- the encoder at the encoding end converts the target image from RGB to YUV, and then performs block, DCT, quantization, run-length encoding, entropy encoding, packing, and marking in turn; then it is sent to the receiving end, the decoder of the receiving end Decode the first code stream to obtain YUV according to the reverse process of encoding, and then convert it from YUV to RGB to obtain the reference frame.
- Step S307 reconstruct the target image according to the reference frame image.
- the decoder 111 will reconstruct the target image according to the reference frame image.
- the target image is any one of the multiple frames of target images included in the target video;
- the reconstructing the target image according to the reference frame image includes: according to the reference frame image and the target image At least one of the reference frame image and the non-reference frame image corresponding to an adjacent frame of target image is reconstructed by using an interpolation algorithm. For example: when performing image encoding, a time packet can be marked for the code stream; when the first code stream is received, the second code stream has not been received within the time period specified by the time packet, and it can be regarded as the second code stream. The code stream is lost.
- the reconstructed frame that is, the target image
- the decoding end receives the first code stream, it can avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, reduce the phenomenon of no sound and freeze in the video information, and improve the user's visual experience.
- the decoder 111 receives a second code stream sent by the encoding end within a preset time period after receiving the first code stream, where the second code stream is a non-reference frame image for performing the processing.
- the code stream obtained after encoding the image, the non-reference frame image includes the remaining images except the reference frame image obtained after down-sampling the target image; if the second code stream is incomplete, the After image decoding is performed on the second bitstream, the corresponding incomplete non-reference frame image is obtained; the peripheral pixels of the incomplete non-reference frame image are determined according to the incomplete non-reference frame image; the reconstruction is based on the reference frame image
- the target image includes: reconstructing the target image through an interpolation algorithm according to peripheral pixels of the reference frame image and the incomplete non-reference frame image.
- the second code stream is received, and if not, it is determined according to the reference frame image corresponding to the first code stream and the target image of the target image. At least one of the reference frame image and the non-reference frame image corresponding to the next frame of the target image is reconstructed; if the second code stream is received, it can be determined whether the received second code stream is incomplete. If the second bit stream is incomplete, the target image can be obtained by interpolating the incomplete second bit stream and the complete first bit stream, which avoids the inability of the receiving end to reconstruct the complete video image due to partial information loss, and reduces the appearance of the video information. Sounds, freezes, and other phenomena enhance the user’s perception and experience.
- a time packet can be marked for the code stream; when the first code stream is received, the second code stream is received within the time period specified by the time packet, and the second code stream needs to be determined at this time Whether the stream is incomplete, if it is determined that the second code stream is incomplete, the target image can be reconstructed according to the information of the incomplete second code stream by using an interpolation algorithm and the complete first code stream information; or it can be based on the information of the incomplete second code stream At least one of a reference frame image and a non-reference frame image corresponding to the complete first code stream information and an adjacent frame of the target image of the target image is used to reconstruct the target image using an interpolation algorithm.
- the decoder 111 receives the second code stream sent by the encoding end within a preset time period after receiving the first code stream; if the second code stream is complete, the second code stream is After image decoding of the stream, the corresponding non-reference frame image is obtained; the reconstruction of the target image according to the reference frame image includes: stitching the reference frame image and the non-reference frame image to Reconstruct the target image.
- a time packet can be marked on the code stream; when the first code stream is received, the second code stream is received within the time period specified by the time packet, if the second code stream is determined If the stream is complete, the reference frame image decoded by the first code stream and the non-reference frame image decoded by the second code stream can be directly spliced into the target image.
- the image coding and decoding algorithms in the embodiments of the present application can quickly track and adapt to changes in the channel environment of the wireless channel, so that the delay and image quality of the receiving end can be maintained at an acceptable level.
- the encoding end adjusts the encoding parameters during image encoding according to the channel information of the current wireless channel, so that the code rate of the generated code stream is lower than the channel capacity of the current wireless channel, and the code stream will not be affected by the channel during wireless transmission.
- the capacity is too low to transmit the complete code stream, or the channel capacity is jittered, or the channel capacity is reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate ) Below, and part of the code stream information is lost.
- the code stream received by the encoding end is obtained by down-sampling the target image and then image encoding. Also, because downsampling can divide the target image into multiple copies of lower resolution, the size of the code stream obtained by down-sampling and re-encoding the target image is smaller than the code stream size obtained when the target image is directly encoded.
- the down-sampled code stream may better adapt to changes in the wireless channel, and reduce the probability of information loss during transmission, and at the same time avoid the inability of the receiving end to reconstruct a complete video image due to information loss.
- the adjustment of transmission parameters and the allocation of channel resources also reduce the probability of information loss during transmission, avoid the inability of the receiving end to reconstruct the complete video image due to information loss, and reduce the occurrence of mosaic, stuttering, and blurring of video information. Phenomenon to enhance the user’s viewing experience.
- FIG. 4A is a schematic structural diagram of a low-delay source-channel joint coding apparatus provided by an embodiment of the present application.
- the low-delay source-channel joint coding apparatus 10 may include a first sampling unit 401 and a first coding unit.
- the unit 402 and the first sending unit 403 may also include: a first encoding parameter unit 404, a first acquiring unit 405, a second sending unit 406, a second acquiring unit 407, and a third sending unit 408. Detailed description of each unit as follows.
- the first sampling unit 401 is used for down-sampling the target image to obtain a reference frame image and a non-reference frame image.
- the first encoding unit 402 is configured to perform image encoding on the reference frame image and the non-reference frame image to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image.
- the first sending unit 403 is configured to determine the first channel resource and the second channel resource based on the channel environment of the current wireless channel, and respectively use the first channel resource to send the first code stream and the second channel resource to send the second code stream, Among them, the first channel resource is better than the second channel resource.
- the above-mentioned first encoding unit 402 is specifically configured to perform intra-frame compression on each of the above-mentioned reference frame images included in the above-mentioned reference frame image to obtain the above-mentioned first code stream;
- the residual of the reference frame image is subjected to the intra-frame compression to obtain the corresponding second code stream.
- the foregoing channel environment further includes channel capacity;
- the device further includes: a first encoding parameter unit 404, configured to perform image encoding on the foregoing reference frame image and the foregoing non-reference frame image separately, Before obtaining the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image, based on the channel capacity, the encoding parameters corresponding to the reference frame image and the non-reference frame image are determined, wherein The above coding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate during image encoding.
- the target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity .
- the above-mentioned channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate;
- the unit 403 is specifically configured to determine the transmission parameters corresponding to the first code stream and/or the second code stream respectively based on the current channel environment, where the transmission parameters are used for target modulation and coding strategy information and/or Send the code stream at the target transmission power, the transmission parameters of the first code stream are better than the transmission parameters of the second code stream; send the first code stream according to the transmission parameters corresponding to the first code stream, and according to the second code stream The corresponding sending parameter sends the above-mentioned second code stream.
- the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the first sending unit 403 is specifically configured to: determine the above-mentioned first wireless channel and the above-mentioned second wireless channel based on the current channel environment.
- the first code stream is sent through a first wireless channel
- the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned apparatus further includes: a first obtaining unit 405, configured to obtain the multi-frame target images included in the above-mentioned target video Each frame of target image corresponds to the code stream of the aforementioned reference frame image; the audio information in the aforementioned target video is acquired; the second sending unit 406 is configured to correspond each frame of the aforementioned target video to the aforementioned reference frame image. The code stream and the audio information are sent through the first wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned apparatus further includes: a second acquisition unit 407, configured to acquire the multi-frame target images included in the above-mentioned target video Each frame of the target image corresponds to the code stream of the aforementioned non-reference frame image; the third sending unit 408 is configured to pass the code stream of each frame image corresponding to the aforementioned non-reference frame image among the multiple frames of images included in the aforementioned target video through the aforementioned second Wireless channel transmission.
- each functional unit in the low-delay source-channel joint coding device 20 described in the embodiment of the present application can be referred to the relevant description in the method embodiment described in FIG. 2A-2C. Here, No longer.
- FIG. 4B is a schematic structural diagram of another low-delay source-channel joint coding apparatus provided by an embodiment of the present application.
- the low-delay source-channel joint coding apparatus 20 may include a second sampling unit 411 and a second sampling unit 411.
- the encoding parameter unit 412, the second encoding unit 413, and the fourth sending unit 414 may also include: a third acquiring unit 415, a fifth sending unit 416, a fourth acquiring unit 417, and a sixth sending unit 418.
- the details of each unit Described as follows.
- the second sampling unit 411 is used for down-sampling the target image to obtain a reference frame image and a non-reference frame image.
- the second encoding parameter unit 412 is configured to determine the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel.
- the second encoding unit 413 is configured to perform image encoding on the reference frame image and the non-reference frame image according to the encoding parameters, to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded .
- the fourth sending unit 414 is configured to send the first code stream and the second code stream respectively.
- the foregoing encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target bit rate during image encoding;
- the foregoing second encoding unit 413 is specifically configured to: include the foregoing reference frame image Each of the aforementioned reference frame images is intra-compressed according to the aforementioned target code rate to obtain the aforementioned first code stream; the residual difference between the aforementioned non-reference frame image and the aforementioned reference frame image is subjected to the aforementioned intra-frame compression according to the aforementioned target code rate to obtain Corresponding to the above second code stream.
- the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate;
- the fourth The sending unit 414 is configured to separately send the first code stream and the second code stream, and is further configured to: determine the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein, the transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the transmission parameters of the first code stream are better than the transmission parameters of the second code stream.
- the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned fourth sending unit 414 is specifically configured to: send the above-mentioned first code stream through the first wireless channel, and transmit the above-mentioned first code stream to the The two code streams are sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned apparatus further includes: a third acquisition unit 415, configured to acquire the multi-frame target images included in the above-mentioned target video Each frame of target image corresponds to the code stream of the aforementioned reference frame image; the audio information in the aforementioned target video is obtained; the fifth sending unit 416 is configured to correspond each frame of the aforementioned target video to the aforementioned reference frame image. The code stream and the audio information are sent through the first wireless channel.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a fourth obtaining unit 417, configured to obtain the multi-frame target images included in the above-mentioned target video Each frame of the target image corresponds to the code stream of the aforementioned non-reference frame image; the sixth sending unit 418 is configured to pass the code stream of each frame image corresponding to the aforementioned non-reference frame image in the multi-frame image included in the aforementioned target video through the aforementioned second Wireless channel transmission.
- each functional unit in the low-delay source-channel joint coding apparatus 20 described in the embodiment of the present application can be referred to the related description in the method embodiment described in FIG. 3A-FIG. 3C, here No longer.
- FIG. 4C is a schematic structural diagram of a low-delay source-channel joint decoding apparatus provided by an embodiment of the present application.
- the low-delay source-channel joint decoding apparatus 30 may include a receiving unit 421, a decoding unit 422, and an image.
- the unit 423 may further include: a fifth acquiring unit 424 and a sixth acquiring unit 425, wherein the detailed description of each unit is as follows.
- the receiving unit 421 is configured to receive a first code stream sent by an encoding end, the first code stream is a code stream obtained after image encoding of a reference frame image, wherein the reference frame image includes a code stream obtained by down-sampling the target image Reference frame image.
- the decoding unit 422 is configured to decode the image of the first code stream to obtain the reference frame image corresponding to the first code stream.
- the image unit 423 is configured to reconstruct the target image according to the reference frame image.
- the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned image unit 423 is specifically configured to: according to the adjacent one of the reference frame image and the target image At least one of the reference frame image and the non-reference frame image corresponding to the frame target image is reconstructed by using an interpolation algorithm.
- the device further includes: a fifth acquiring unit 424, configured to receive the second code stream sent by the encoding end within a preset time period after receiving the first code stream, where
- the second code stream is a code stream obtained by encoding a non-reference frame image, and the non-reference frame image includes images obtained after down-sampling the target image except for the reference frame image; if the first The second code stream is incomplete.
- the image unit 423 is specifically configured to: according to the peripheral pixels of the above-mentioned reference frame image and the above-mentioned incomplete non-reference frame image, The above-mentioned target image is reconstructed by interpolation algorithm.
- the device further includes: a sixth obtaining unit 425, configured to, if the second code stream is complete, perform image decoding on the second code stream to obtain the corresponding non-reference frame image
- the image unit 423 is specifically configured to: stitch the reference frame image and the non-reference frame image to reconstruct the target image.
- each functional unit in the low-delay source-channel joint coding apparatus 20 described in the embodiment of the present application can be referred to the related description in the method embodiment described in FIG. 3A-FIG. 3C, here No longer.
- FIG. 5 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
- the terminal device 40 is an intelligent terminal capable of wireless image transmission, such as a projector, a video recorder, a mobile phone, a tablet computer, and a vehicle-mounted terminal.
- the device includes at least one processor 501, at least one memory 502, and at least one communication interface 503.
- the device may also include general components such as antennas, which will not be described in detail here.
- the processor 501 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gata Array, FPGA) circuit, or one or more An integrated circuit used to control the execution of the above program program.
- CPU central processing unit
- ASIC application-specific integrated circuit
- FPGA field programmable gate array
- the communication interface 503 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), core network, wireless local area networks (WLAN), etc.
- RAN radio access network
- WLAN wireless local area networks
- the memory 502 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types that can store information and instructions
- the dynamic storage device can also be electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disc storage, optical disc storage (Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures and can be used by a computer Any other media accessed, but not limited to this.
- the memory can exist independently and is connected to the processor through a bus.
- the memory can also be integrated with the processor.
- the memory 502 is used to store application program codes for executing the above solutions, and the processor 501 controls the execution.
- the processor 501 is configured to execute application program codes stored in the memory 502.
- the code stored in the memory 502 can execute the low-delay source-channel joint coding method provided in Figure 2A, such as: down-sampling the target image to obtain a reference frame image and a non-reference frame image; separate the reference frame image and the non-reference frame image Perform image encoding to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image; based on the channel environment of the current wireless channel, determine the first channel resource and the second channel resource, and use them respectively The first channel resource is used to send the first code stream and the second channel resource is used to send the second code stream, wherein the first channel resource is better than the second channel resource.
- the low-delay source-channel joint coding method provided in Figure 2A, such as: down-sampling the target image to obtain a reference frame image and a non-reference frame image; separate the reference frame image and the non-reference frame image Perform image encoding to obtain the first code stream after encoding the reference frame image and the
- the code stored in the memory 502 can also execute the low-latency source-channel joint coding method provided in FIG. 3A, such as: down-sampling the target image to obtain the reference frame image and the non-reference frame image; according to the current wireless channel channel environment, determine Encoding parameters corresponding to the reference frame image and the non-reference frame image; according to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain the encoded reference frame image
- the first code stream and the second code stream after encoding the non-reference frame image; the first code stream and the second code stream are sent separately.
- the disclosed device may be implemented in other ways.
- the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented.
- the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.
- the units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
- the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
- the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
- the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
- the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to enable a computer device (which may be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to execute all or part of the steps of the above methods of the various embodiments of the present application.
- the aforementioned storage media may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviation: ROM) or Random Access Memory (Random Access Memory, abbreviation: RAM), etc.
- U disk mobile hard disk
- magnetic disk magnetic disk
- optical disk read-only memory
- Read-Only Memory abbreviation: ROM
- Random Access Memory Random Access Memory
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
本申请实施例公开了一种低延迟信源信道联合编码方法及相关设备,其中,一种低延迟信源信道联合编码方法包括:首先对目标图像进行下采样,得到基准帧图像与非基准帧图像;对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;基于当前无线信道的信道环境,利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流,其中,第一信道资源优于第二信道资源。实施本申请实施例可以在信道容量降低时,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
Description
本申请涉及无线传输技术领域,尤其涉及一种低延迟信源信道联合编码方法及相关设备。
随着无线传输技术的发展,令使用无线网络承载高清视频成为可能,但同时无线网络承载高清视频也对图像编解码技术提出了新的要求,因为在无线传输时,无线信道短时时变的特性会导致信道容量急剧变化,所以图像编解码算法需要能快速跟踪并适应这一变化,令接收端的延迟和图像质量保持在可接受的水平。
而如联合图像专家组(Joint Photographic Experts Group,JPEG)标准编解码和动态图像专家组(Moving Picture Experts Group,MPEG)标准编解码,这些着眼于压缩倍数和图像质量的传统图像编解码算法,并不能及时快速的根据信道容量做出相应的编码调整。因此,当信道容量降低至信源码率以下时,利用传统图像编解码算法压缩后的视频信息在无线传输的过程中很容易丢失相关码流信息,导致解码后的整个宏块将无法在接收端重构,进而会造成传输过去的视频会出现不同程度的马赛克,卡顿、模糊等现象,观感很差。
因此,如何在传输过程中减少信息丢失,避免由于信息丢失导致接收端无法重构完整的视频图像,是亟待解决的问题。
发明内容
本申请实施例提供一种低延迟信源信道联合编码方法及相关设备,通过对图像的编码或传输方式进行优化,可以在视频传输过程中减少信息丢失,以避免接收端无法播放清晰完整视频,提升用户观看体验。
第一方面,本申请实施例提供了一种低延迟信源信道联合编码方法方法,可包括:对目标图像进行下采样,得到基准帧图像与非基准帧图像;对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流,其中,第一信道资源优于第二信道资源。
视频无线传输技术主要包括视频编码、无线传输、视频解码与显示等几个主要环节。通过第一方面所述方法,在视频编码和无线传输时,编码端可以对目标图像进行下采样,获得基准帧图像与非基准帧图像,然后对基准帧图像进行图像编码获得第一码流,对非基准帧图像进行图像编码获得第二码流,最后利用第一信道资源发送第一码流,以及利用第二信道资源发送第二码流。其中,第一信道资源以及第二信道资源根据当前无线信道的信道环境确定,且第一信道资源优于第二信道资源,即可以理解为,在无线传输的过程中,可以将基准帧图像编码后的第一码流与非基准帧图像编码后的第二码流分级发送,例如:可以设置第一码流的优先级高于第二码流的优先级,然后在分别发送第一码流与第二码流 时,根据当前无线信道的信道环境,使得在发送第一码流时信道的服务质量高于发送第二码流时信道的服务质量。或者,当前无线信道的信道环境较差时,编码端可以优先发送比较重要的第一码流,再发送级别比较低的第二码流;或者,当前无线信道的信道环境较差时,在接收端接收到第一码流后或者在发送第一码流后的一段时间后,再发送级别比较低的第二码流。基于当前无线信道的信道环境,编码端分别发送第一码流和第二码流,可以使得接收端在无法接收到完整的码流时,提高接收端接收到第一码流的概率,利用接收到的第一码流得到完整的重构图像。其次,在对目标图像进行图像编码前,先对目标图像进行下采样,在获得多个分辨率较低的副本图像后,再进行图像编码,可以使得进行下采样后的获得码流要小于直接对目标图像进行图像编码时获得的码流大小,进而更容易使得接收端的解码器接收到完整的码流。因此,编码端对目标图像通过下采样进行图像编码后,将编码后的码流信息基于当前无线信道的信道环境,分别通过无线信道发送,可以使得编码后的码流信息仍可以适应实时变化的信道环境,进而降低在无线传输过程中信息丢失的概率,进而避免由于信息丢失导致接收端无法重构完整的视频图像,提升用户的观看体验。
在一种可能实现的方式中,上述对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流,包括:将上述基准帧图像包括的每个上述基准帧图像进行帧内压缩获得上述第一码流;将上述非基准帧图像与上述基准帧图像的残差进行上述帧内压缩获得对应的上述第二码流。实施本申请实施例时,编码端可以在选择较为简便的下采样将目标图像分层,降低分辨率的同时,对分层后的副本图像进行图像编码时使用的帧内压缩编码方法,由于帧内压缩是空域压缩,当压缩一帧图像时,仅考虑本帧的数据而不考虑相邻帧之间的冗余信息,而帧间压缩要参考其他帧数据,压缩率大,因此,考虑到图像延迟方面,本申请实施例对下采样后的副本图像进行图像编码时并不使用帧间压缩。而且,使用帧内压缩的方法可以在信道容量发生抖动,并降低至初始码率以下时,可以使得接收端的解码器能够接受到基于较小分辨率副本图像图像编码后的码流信息,能够根据该码流信息得到的重构图像。同时在信道容量降低时,减小传输过程中信息丢失的概率,避免视频图像信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在一种可能实现的方式中,上述信道环境还包括信道容量;上述对上述基准帧图像和上述非基准帧图像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基准帧图像编码后的第二码流之前,还包括:基于上述信道容量,确定上述基准帧图像和上述非基准帧图像对应的编码参数,其中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流,上述第一码流和上述第二码流分别对应的上述目标码率均小于或等于上述信道容量。实施本申请实施例时,编码端可以在当信道容量发生变化时,通过发射机获取当前信道的信道容量,根据该反馈回的信道容量信息调整编码器在对副本图像进行图像编码时的编码参数,使得编码后的码流的码率可以在适应于当前的信道容量的情况下,能够顺利发送目标码流,减少因为信道容量降低,造成码流丢失的情况。同时也降低因无法传输完整码流,使得接收端的解码器不能够根据该缺失码流信息得到的重构图像,降低了视频图像卡顿、模糊的概率,提升了用户的观看体验。
在一种可能实现的方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信 噪比、接收的信号强度指示、占空比、比特率中的一个或多个;上述基于当前的信道环境,确定第一信道资源以及第二信道资源,分别利用上述第一信道资源发送上述第一码流以及利用上述第二信道资源发送上述第二码流,包括:基于当前的上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数;按照上述第一码流对应的发送参数发送上述第一码流,以及按照上述第二码流对应的发送参数发送上述第二码流。实施本申请实施例时,编码端可以在当信道发生变化时,通过发射机获取当前无线信道的信道环境(如:信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率等),根据该信道环境调整在发送第一码流和/或发送第二码流时的发送参数(如:发射功率、调制与编码策略信息等),可以理解的是,根据信道信息调整发送参数后,第一码流的发送参数与第二码流的发送参数不同,第一码流的发送参数与第二码流的发送参数相比能够使发射机以更迅速的发送速度、更高的服务质量(Quality of Service,QoS)发送第一码流。因此,在无线传输的过程中,将上述第一码流利用信道资源优于上述第二码流的信道资源发送,即,发送第一码流时的传输质量高于发送第二码流时的传输质量,或者,优先发送比较重要码流(即,副本图像数量少的进行图像编码后的码流),使得在信道容量等发生抖动,造成接收端的解码器无法接收到完整的码流时,因为帧内压缩的特性,可以避免由于信息丢失导致接收端无法重构完整的视频图像,使解码器仅仅基于优先接收到的部分码流得到的重构图像,可以降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在一种可能实现的方式中,上述无线信道包括第一无线信道和第二无线信道;上述基于当前的信道环境,确定第一信道资源以及第二信道资源,分别利用上述第一信道资源发送上述第一码流以及利用上述第二信道资源发送上述第二码流,包括:基于当前的信道环境,确定上述第一无线信道和上述第二无线信道,将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。实施本申请实施例时,编码端可以在无线传输的过程中,将视频源分级,上述第一码流利用信道资源优于上述第二码流的信道资源发送。例如:将第一码流映射到视频(Video,VI)业务,将第二码流映射到尽力(Best Effort,BE)业务。因此,在信道容量等发生抖动时,通过无线信道发送的第一码流和第二码流分别使用不同的QoS保障级别,提高接收端接收到第一码流的概率,使得接收端的解码器可以在仅仅接收到通过服务质量保障机制高的信道发送的第一码流时,仍然可以仅仅通过第一码流重构图像,避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述方法还包括:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。实施本申请实施例时,编码端可以获取目标视频包括的多帧目标图像中所有目标图像对应上述基准帧图像的码流与音频信息一起映射到服务质量保障机制高的VI业务发送,可以理解的是, 上述音频信息可以是通过语音编码后的音频信息。在信道发生变化时,接收端的解码器可以仅仅基于通过服务质量保障机制高的信道接收到的所有目标图像对应的码流和音频信息重构视频,提高接收端接收到第一码流的概率,避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述方法还包括:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。实施本申请实施例时,编码端可以在获取目标视频包括的多帧目标图像中所有目标图像对应上述非基准帧图像的码流后,单独映射到服务质量保障机制稍低的BE业务发送。因此,即使所有目标图像的非基准帧图像对应的码流在信道发生变化时丢失部分甚至是全部码流信息时,解码端可以仅仅基于通过服务质量保障机制高的信道接收到的所有目标图像的第一副本对应的码流和音频信息重构视频,避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
第二方面,本申请实施例提供了另一种低延迟信源信道联合编码方法,可包括:对目标图像进行下采样,得到基准帧图像与非基准帧图像;根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数;根据编码参数,对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;分别发送第一码流以及第二码流。
实施本申请实施例,编码端可以首先对目标图像进行下采样,获得基准帧图像和非基准帧图像;然后根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数,再利用该编码参数对基准帧图像和非基准帧图像进行图像编码,使得第一码流和第二码流的码率低于当前无线信道的信道容量;最后再将第一码流和第二码流分别通过无线信道发送出去。其中,根据当前无线信道的信道信息,调整图像编码时的编码参数,可以令生成的码流的码率低于当前无线信道的信道容量,进而码流在无线传输时,不会因为信道容量低至无法传输完整码流,或者信道容量发生抖动,又或者信道容量降低至初始码率(如:初始码率可以为按照传统编码方法对目标图像进行压缩时的码率或者预设码率)以下时,而丢失部分码流信息。因此,本申请实施例的图像编解码算法能快速跟踪并适应无线信道的信道环境的变化,令接收端的延迟和图像质量保持在可接受的水平。其次,编码端在对目标图像进行图像编码前,先对目标图像进行下采样,在获得多个分辨率较低的副本图像后,再进行图像编码,可以使得进行下采样后的获得码流要小于直接对目标图像进行图像编码时获得的码流大小,进而更容易使得接收端的解码器接收到完整的码流。因此,编码端根据当前无线信道的信道信息进行图像编码获得的码流信息,可以适应实时变化的信道环境,进而降低在传输过程中信息丢失的概率,避免由于信息丢失导致接收端无法重构完整的视频图像,提升用户的观看体验。
在一种可能的实现方式中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流;上述根据上述编码参数,对上述基准帧图像和上述非基准帧图 像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基准帧图像编码后的第二码流,包括:将上述基准帧图像包括的每个上述基准帧图像按照上述目标码率进行帧内压缩,获得上述第一码流;将上述非基准帧图像中包含的每个上述非基准帧图像与上述基准帧图像包括的所有上述基准帧图像的残差按照上述目标码率进行上述帧内压缩,获得对应的上述第二码流。实施本申请实施例时,编码端可以在选择较为简便的下采样将目标图像分层,降低分辨率的同时,对分层后的副本图像进行图像编码时使用的帧内压缩编码方法,由于帧内压缩是空域压缩,当压缩一帧图像时,仅考虑本帧的数据而不考虑相邻帧之间的冗余信息,而帧间压缩要参考其他帧数据,压缩率大,因此,考虑到图像延迟方面,编码端对下采样后的副本图像使用帧内压缩的方法可以在信道容量发生抖动,并降低至初始码率以下时,可以使得接收端的解码器能够接受到基于较小分辨率副本图像图像编码后的码流信息,能够根据该码流信息得到的重构图像。同时在信道容量降低时,减小传输过程中信息丢失的概率,避免视频图像信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在一种可能的实现方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;上述分别发送上述第一码流以及上述第二码流之前,还包括:根据上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数。实施本申请实施例时,编码端可以在当信道发生变化时,通过发射机获取当前无线信道的信道环境,根据该信道环境调整在发送第一码流和/或发送第二码流时的发送参数(如:发射功率、调制与编码策略信息等),可以理解的是,根据信道信息调整发送参数后,第一码流的发送参数与第二码流的发送参数不同,第一码流的发送参数与第二码流的发送参数相比能够使发射机以更迅速的发送速度、更高的服务质量(Quality of Service,QoS)发送第一码流。因此,在无线传输的过程中,编码端将上述第一码流利用信道资源优于上述第二码流的信道资源发送,即,发送第一码流时的传输质量高于发送第二码流时的传输质量,或者,优先发送比较重要码流(即,副本图像数量少进行图像编码后的码流),使得在信道容量等发生抖动时,接收端的解码器仍然可以接收到完整的码流,避免信息丢失,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在一种可能的实现方式中,上述无线信道包括第一无线信道和第二无线信道;上述分别发送上述第一码流以及上述第二码流,包括:将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。实施本申请实施例时,编码端可以在无线传输的过程中,将视频源分级,上述第一码流利用信道资源优于上述第二码流的信道资源发送。因此,在信道容量等发生抖动时,编码端通过无线信道发送的第一码流和第二码流分别使用不同的QoS保障级别,提高接收端接收到第一码流的概率,使得接收端的解码器可以在仅仅接收到通过服务质量保障机制高的信道发送的第一码流时,仍然可以仅仅通过第一码流重构图像,避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述方法还包括:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。实施本申请实施例时,编码端可以获取目标视频包括的多帧目标图像中所有目标图像对应上述基准帧图像的码流与音频信息一起映射到服务质量保障机制高的VI业务发送,可以理解的是,上述音频信息可以是通过语音编码后的音频信息。在信道发生变化时,接收端的解码器可以仅仅基于通过服务质量保障机制高的信道接收到的所有目标图像对应的码流和音频信息重构视频,提高接收端接收到第一码流的概率,避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述方法还包括:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。实施本申请实施例时,编码端可以在获取目标视频包括的多帧目标图像中所有目标图像对应上述非基准帧图像的码流后,单独映射到服务质量保障机制稍低的BE业务发送。因此,即使所有目标图像的非基准帧图像对应的码流在信道发生变化时丢失部分甚至是全部码流信息时,解码端可以仅仅基于通过服务质量保障机制高的信道接收到的所有目标图像的第一副本对应的码流和音频信息重构视频,避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
第三方面,本申请实施例提供了一种低延迟信源信道联合解码方法,可包括:接收编码端发送的第一码流,第一码流为基准帧图像进行图像编码后获得的码流,其中,基准帧图像包括一个或多个对目标图像进行下采样后获得的图像;对第一码流进行图像解码后,获得第一码流对应的基准帧图像;根据基准帧图像,重构目标图像。
在无线传输的过程中,通过本申请第三方面提供的解码方法,解码端(即,接收端)可以接收编码端发送的第一码流,进而将第一码流进行图像解码后,获得第一码流对应的基准帧图像,然后根据基准帧图像,重构目标图像。需要说明的是,第一码流为基准帧图像进行图像编码后获得的码流,其中,基准帧图像包括一个或多个对目标图像进行下采样后获得的图像,因此,接收端接收的码流是对目标图像进行下采样再图像编码获得的。又由于下采样可以将目标图像采样为多个分辨率较低的副本图像,可以将其中一部分作为基准帧图像。而对目标图像进行下采样再编码获得的码流大小要小于直接对目标图像进行图像编码时获得的码流大小,因此,目标图像下采样后的码流可能会更好适应无线信道的变化,并在传输过程中减少信息丢失的概率,同时也避免了由于信息丢失导致接收端无法重构完整的视频图像。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述根据上述基准帧图像,重构上述目标图像,包括:根据上述基准帧图像与上述目标图像相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,通过插值算 法重构上述目标图像。实施本申请实施例,接收端可以在接收端仅仅接收到第一码流时,利用插值算法将第一码流对应的基准帧图像与第一码流对应的目标图像的前一帧目标图像对应的基准帧图像重构成该目标图像,因此,当接收端接收到第一码流时,即可以避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。
在一种可能的实现方式中,所述方法还包括:在接收上述第一码流后的预设时间段内,接收上述编码端发送的第二码流,其中,上述第二码流为非基准帧图像进行上述图像编码后获得的码流,上述非基准帧图像包括对上述目标图像进行下采样后获得的除上述基准帧图像外剩余的图像;若上述第二码流残缺,将上述第二码流进行图像解码后,获得对应的残缺非基准帧图像;根据上述残缺非基准帧图像,确定上述残缺非基准帧图像的周边像素;上述根据上述基准帧图像,重构上述目标图像,包括:根据上述基准帧图像与上述残缺非基准帧图像的周边像素,通过插值算法重构上述目标图像。实施本申请实施例,接收端可以在接受到第一码流的预设时间段内,确定是否接收到第二码流,若否,则根据第一码流与上述相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,重构目标图像;若接收到了第二码流,则确定接收到的上述第二码流是否残缺,若接收到的第二码流残缺则可以利用残缺的第二码流与完整的第一码流插值得到目标图像,避免了由于部分信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。
在一种可能的实现方式中,所述方法还包括:若确定上述第二码流完整,将上述第二码流进行图像解码后,获得对应的上述非基准帧图像;上述根据上述基准帧图像,重构上述目标图像,包括:将上述基准帧图像和上述非基准帧图像拼接,以重构上述目标图像。实施本申请实施例,接收端可以在接收到完整的第一码流和第二码流时,直接进行图像解码后拼接为图像,适应了无线信道的变化,并在传输过程中信息保存完整,避免由于信息丢失导致接收端无法重构完整的视频图像,提升了用户的观看体验。
第四方面,本申请实施例提供了一种低延迟信源信道联合编码装置,其特征在于,包括:编码器和发射机,其中,编码器,用于:对目标图像进行下采样,得到基准帧图像与非基准帧图像;对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流;发射机,用于:基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用所述第一信道资源发送所述第一码流以及利用所述第二信道资源发送所述第二码流,其中,所述第一信道资源优于所述第二信道资源。
在一种可能实现的方式中,上述编码器,用于将上述基准帧图像和上述非基准帧图像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基准帧图像编码后的第二码流时,上述编码器具体用于:将上述基准帧图像包括的每个上述基准帧图像进行帧内压缩获得上述第一码流;将上述非基准帧图像与上述基准帧图像的残差进行上述帧内压缩获得对应的上述第二码流。
在一种可能实现的方式中,上述信道环境还包括信道容量;上述发射机,在上述对上 述基准帧图像和上述非基准帧图像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基准帧图像编码后的第二码流之前,还用于:基于上述信道容量,确定上述基准帧图像和上述非基准帧图像对应的编码参数,其中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流,上述第一码流和上述第二码流分别对应的上述目标码率均小于或等于上述信道容量。
在一种可能实现的方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;上述发射机,在用于上述基于当前的信道环境,确定第一信道资源以及第二信道资源,分别利用上述第一信道资源发送上述第一码流以及利用上述第二信道资源发送上述第二码流时,具体用于:基于当前的上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数;按照上述第一码流对应的发送参数发送上述第一码流,以及按照上述第二码流对应的发送参数发送上述第二码流。
在一种可能实现的方式中,上述无线信道包括第一无线信道和第二无线信道;上述发射机,在用于上述基于当前的信道环境,确定第一信道资源以及第二信道资源,分别利用上述第一信道资源发送上述第一码流以及利用上述第二信道资源发送上述第二码流时,具体用于:基于当前的信道环境,确定上述第一无线信道和上述第二无线信道,将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述编码器还用于:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;上述发射机还用于:将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述编码器还用于:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;上述发射机还用于:将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。
第五方面,本申请实施例提供了另一种低延迟信源信道联合编码装置,其特征在于,包括:编码器和发射机,其中,编码器,用于:对目标图像进行下采样,得到基准帧图像与非基准帧图像;根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数;根据编码参数,对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;发射机,用于:分别发送第一码流以及第二码流。
在一种可能的实现方式中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流;编码器,用于上述根据上述编码参数,对上述基准帧图像和上述非基准帧图像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基 准帧图像编码后的第二码流时,具体用于:将上述基准帧图像包括的每个上述基准帧图像按照上述目标码率进行帧内压缩,获得上述第一码流;将上述非基准帧图像与上述基准帧图像的残差按照上述目标码率进行上述帧内压缩,获得对应的上述第二码流。
在一种可能的实现方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;上述发射机,用于分别发送上述第一码流以及上述第二码流之前,还用于:根据上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数。
在一种可能的实现方式中,上述无线信道包括第一无线信道和第二无线信道;上述发射机,用于分别发送上述第一码流以及上述第二码流时,具体用于:将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述编码器还用于:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;上述发射机还用于:将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述编码器还用于:获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;上述发射机还用于:将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。
第六方面,本申请实施例提供了一种低延迟信源信道联合编码装置,其特征在于,包括:第一采样单元,用于对目标图像进行下采样,得到基准帧图像与非基准帧图像;第一编码单元,用于对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;第一发送单元,用于基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流,其中,第一信道资源优于第二信道资源。
在一种可能实现的方式中,上述第一编码单元,具体用于将上述基准帧图像包括的每个上述基准帧图像进行帧内压缩获得上述第一码流;将上述非基准帧图像中包括的每个上述非基准帧图像与上述基准帧图像包括的所有的上述基准帧图像的残差进行上述帧内压缩获得对应的上述第二码流。
在一种可能实现的方式中,上述信道环境还包括信道容量;所述装置还包括:第一编码参数单元,用于在上述对上述基准帧图像和上述非基准帧图像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基准帧图像编码后的第二码流之前,基于上述信道容量,确定上述基准帧图像和上述非基准帧图像对应的编码参数,其中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流,上述第一码流 和上述第二码流分别对应的上述目标码率均小于或等于上述信道容量。
在一种可能实现的方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;第一发送单元,具体用于基于当前的上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数;按照上述第一码流对应的发送参数发送上述第一码流,以及按照上述第二码流对应的发送参数发送上述第二码流。
在一种可能实现的方式中,上述无线信道包括第一无线信道和第二无线信道;第一发送单元,具体用于:基于当前的信道环境,确定上述第一无线信道和上述第二无线信道,将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第一获取单元,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;第二发送单元,用于将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第二获取单元,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;第三发送单元,用于将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。
第七方面,本申请实施例提供了另一种低延迟信源信道联合编码装置,其特征在于,包括:第二采样单元,用于对目标图像进行下采样,得到基准帧图像与非基准帧图像;第二编码参数单元,用于根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数;第二编码单元,用于根据编码参数,对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;第四发送单元,用于分别发送第一码流以及第二码流。
在一种可能的实现方式中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流;上述第二编码单元,具体用于:将上述基准帧图像包括的每个上述基准帧图像按照上述目标码率进行帧内压缩,获得上述第一码流;将上述非基准帧图像与上述基准帧图像的残差按照上述目标码率进行上述帧内压缩,获得对应的上述第二码流。
在一种可能的实现方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;上述第四发送单元,用于分别发送上述第一码流以及上述第二码流之前,还用于:根据上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数。
在一种可能的实现方式中,上述无线信道包括第一无线信道和第二无线信道;上述第四发送单元,具体用于:将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第三获取单元,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;第五发送单元,用于将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第四获取单元,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;第六发送单元,用于将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。
第八方面,本申请实施例提供了一种低延迟信源信道联合解码装置,其特征在于,包括:接收单元,用于接收编码端发送的第一码流,第一码流为基准帧图像进行图像编码后获得的码流,其中,基准帧图像包括一个或多个对目标图像进行下采样后获得的图像;解码单元,用于对第一码流进行图像解码后,获得第一码流对应的上述基准帧图像;图像单元,用于根据上述基准帧图像,重构上述目标图像。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述图像单元,具体用于:根据上述基准帧图像与上述目标图像相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,通过插值算法重构上述目标图像。
在一种可能的实现方式中,所述装置还包括:第五获取单元,用于若在接收上述第一码流后的预设时间段内,接收上述编码端发送的第二码流,其中,上述第二码流为非基准帧图像进行上述图像编码后获得的码流,上述非基准帧图像包括对上述目标图像进行下采样后获得的除上述基准帧图像外剩余的图像;若上述第二码流残缺,将上述第二码流进行图像解码后,获得对应的残缺非基准帧图像;根据所述残缺非基准帧图像,确定所述残缺非基准帧图像的周边像素;上述图像单元,具体用于:根据上述基准帧图像与上述残缺非基准帧图像的周边像素,通过插值算法重构上述目标图像。
在一种可能的实现方式中,所述装置还包括:第六获取单元,用于若确定上述第二码流完整,将上述第二码流进行图像解码后,获得对应的上述非基准帧图像;上述图像单元,具体用于:将上述基准帧图像和上述非基准帧图像拼接重构上述目标图像。
第九方面,本申请实施例提供一种服务设备,该服务设备中包括处理器,处理器被配置为支持该服务设备执行上述第一方面提供的一种低延迟信源信道联合编码方法或上述第二方面提供的另一种低延迟信源信道联合编码方法中相应的功能。该服务设备还可以包括存储器,存储器用于与处理器耦合,其保存该服务设备必要的程序指令和数据。该服务设备还可以包括通信接口,用于该服务设备与其他设备或通信网络通信。
第十方面,本申请实施例提供一种服务设备,该服务设备中包括处理器,处理器被配 置为支持该服务设备执行上述第三方面提供的一种低延迟信源信道联合解码方法中相应的功能。该服务设备还可以包括存储器,存储器用于与处理器耦合,其保存该服务设备必要的程序指令和数据。该服务设备还可以包括通信接口,用于该服务设备与其他设备或通信网络通信。
第十一方面,本申请实施例提供了一种计算机程序,该计算机程序包括指令,当该计算机程序被计算机执行时,使得计算机可以执行上述第四方面提供的一种低延迟信源信道联合编码装置或上述第五方面提供的另一种低延迟信源信道联合编码装置所执行的流程。
第十二方面,本申请实施例提供了一种计算机程序,该计算机程序包括指令,当该计算机程序被计算机执行时,使得计算机可以执行上述第六方面提供的一种低延迟信源信道联合解码装置所执行的流程。
第十三方面,本申请实施例提供一种计算机存储介质,用于储存为上述第四方面提供的一种低延迟信源信道联合编码装置,或上述第五方面提供的另一种低延迟信源信道联合编码装置,或上述第六方面提供的一种低延迟信源信道联合解码装置所用的计算机软件指令,其包含用于执行上述方面所设计的程序。
第十四方面,本申请实施例提供了一种芯片***,该芯片***包括处理器,用于支持服务设备实现上述第一方面、第二方面或第三方面中所涉及的功能。在一种可能的设计中,所述芯片***还包括存储器,所述存储器,用于保存数据发送设备必要的程序指令和数据。该芯片***,可以由芯片构成,也可以包含芯片和其他分立器件。
第十五方面,本申请实施例提供一种电子设备,包括上述第一方面、第二方面或第三方面中的任意一种实现方式所提供的处理芯片以及耦合于所述芯片的分立器件。
为了更清楚地说明本申请实施例中的技术方案,下面将对本申请实施例或背景技术中所需要使用的附图进行说明。
图1是本申请实施例提供的一种低延迟信源信道联合编码***构架示意图。
图2A是本申请实施例提供的一种低延迟信源信道联合编码方法的流程示意图。
图2B是本申请提供的一种应用场景示意图。
图2C是本申请实施例提供的一种第一码流和第二码流分别通过第一信道和第二信道发送的示意图。
图3A是本申请实施例提供的另一种低延迟信源信道联合编解码方法的流程示意图。
图3B是本申请实施例提供的一种对目标图像进行编解码应用示意图。
图3C是本申请提供的另一种应用场景示意图。
图4A是本申请实施例提供的一种低延迟信源信道联合编码装置的结构示意图。
图4B是本申请实施例提供的另一种低延迟信源信道联合编码装置的结构示意图。
图4C是本申请实施例提供的一种低延迟信源信道联合解码装置的结构示意图。
图5是本申请实施例提供的一种终端设备的结构示意图。
下面将结合本申请实施例中的附图,对本申请实施例进行描述。
本申请的说明书和权利要求书及所述附图中的术语“第一”、“第二”、“第三”和“第四”等是用于区别不同对象,而不是用于描述特定顺序。此外,术语“包括”和“具有”以及它们任何变形,意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、***、产品或设备没有限定于已列出的步骤或单元,而是可选地还包括没有列出的步骤或单元,或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本文中提及“实施例”意味着,结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例,也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是,本文所描述的实施例可以与其它实施例相结合。
在本说明书中使用的术语“部件”、“模块”、“***”等用于表示计算机相关的实体、硬件、固件、硬件和软件的组合、软件、或执行中的软件。例如,部件可以是但不限于,在处理器上运行的进程、处理器、对象、可执行文件、执行线程、程序和/或计算机。通过图示,在计算设备上运行的应用和计算设备都可以是部件。一个或多个部件可驻留在进程和/或执行线程中,部件可位于一个计算机上和/或分布在2个或更多个计算机之间。此外,这些部件可从在上面存储有各种数据结构的各种计算机可读介质执行。部件可例如根据具有一个或多个数据分组(例如来自与本地***、分布式***和/或网络间的另一部件交互的二个部件的数据,例如,通过信号与其它***交互的互联网)的信号通过本地和/或远程进程来通信。
首先,对本申请中的部分用语进行解释说明,以便于本领域技术人员理解。
(1)联合图像专家组(Joint Photographic Experts Group,JPEG),是一个国际标准化组织(International Standardization Organization,ISO)/国际电工委员会(International Electrotechnical Commission,IEC)的专家组,他们开发并维护计算机图像文件的一套压缩算法的标准。JPEG的压缩方式通常是破坏性数据压缩(lossy compression),意即在压缩过程中图像的质量会遭受到可见的破坏。
(2)动态图像专家组(Moving Picture Experts Group,MPEG),MPEG是ISO与IEC于1988年成立的专门针对运动图像和语音压缩制定国际标准的组织。MPEG标准主要有以下五个,MPEG-1、MPEG-2、MPEG-4、MPEG-7及MPEG-21等。MPEG标准的视频压缩编码技术主要利用了具有运动补偿的帧间压缩编码技术以减小时间冗余度,利用离散余弦变换技术以减小图像的空间冗余度,利用熵编码则在信息表示方面减小了统计冗余度。
(3)RGB图像,R代表red;G代表green;B代表blue,在视觉效果上,都可以用不同比例的:红、绿、蓝来合成,彩色图像的每一个像素,都用红、绿、蓝的不同比例来表示,这样的图像就是RGB图像。
(4)无线局域网(Wireless Local Area Network,WLAN),它是相当便利的数据传输***,它利用射频(Radio Frequency;RF)的技术,使用电磁波,取代旧式碍手碍脚的双绞铜线(Coaxial)所构成的局域网络,在空中进行通信连接,使得无线局域网络能利用简单的存取架构让用户透过它,达到“信息随身化、便利走天下”的理想境界。
(5)离散余弦变换(Discrete Cosine Transform,DCT),是与傅里叶变换相关的一种变换,类似于离散傅里叶变换,但是只使用实数。离散余弦变换相当于一个长度大概是它两倍的离散傅里叶变换,这个离散傅里叶变换是对一个实偶函数进行的(因为一个实偶函数的傅里叶变换仍然是一个实偶函数),在有些变形里面需要将输入或者输出的位置移动半个单位。有两个相关的变换,一个是离散正弦变换,它相当于一个长度大概是它两倍的实奇函数的离散傅里叶变换;另一个是改进的离散余弦变换,它相当于对交叠的数据进行离散余弦变换。
(6)游长编码(Run Length Coding,RLC),又称“运行长度编码”或“行程编码”,是一种统计编码,该编码属于无损压缩编码,是栅格数据压缩的重要编码方法。对于二值图有效。游长编码的基本原理是:用一个符号值或串长代替具有相同值的连续符号(连续符号构成了一段连续的“行程”。行程编码因此而得名),使符号长度少于原始数据的长度。只在各行或者各列数据的代码发生变化时,一次记录该代码及相同代码重复的个数,从而实现数据的压缩。
(7)信号与干扰加噪声比(Signal to Interference plus Noise Ratio,SINR)是指:信号与干扰加噪声比(SINR)是接收到的有用信号的强度与接收到的干扰信号(噪声和干扰)的强度的比值。
(8)信噪比(Signal to Noise Ratio,SNR),又称为讯噪比,是指一个电子设备或者电子***中信号与噪声的比例。这里面的信号指的是来自设备外部需要通过这台设备进行处理的电子信号,噪声是指经过该设备后产生的原信号中并不存在的无规则的额外信号(或信息),并且该种信号并不随原信号的变化而变化。
(9)误包率(Packet Error Ratio,PER),衡量数据分组(数据包)在规定时间内传输准确性的指标,误包率也是衡量数据业务服务质量的重要指标。误包率=传输中的误包数/所传输的总包数*100%。
(10)宏块(Macroblock),是视频编码技术中的一个基本概念。在视频编码中,一个编码图像通常划分成若干宏块组成,一个宏块由一个亮度像素块和附加的两个色度像素块组成。一般来说,亮度块为16x16大小的像素块,而两个色度图像像素块的大小依据其图像的采样格式而定,如:对于YUV420采样图像,色度块为8x8大小的像素块。每个图象中,若干宏块被排列成片的形式,视频编码算法以宏块为单位,逐个宏块进行编码,组织成连续的视频码流。。
(11)码流(Data Rate),是指视频文件在单位时间内使用的数据流量,也叫码率,是视频编码中画面质量控制中最重要的部分。
(12)熵编码,是一种独立于介质的具体特征的进行无损数据压缩的方案。一种主要类型的熵编码方式是对输入的每一个符号,创建并分配一个唯一的前缀码,然后,通过将每个固定长度的输入符号替换成相应的可变长度前缀无关(prefix-free)输出码字替换,从而达到压缩数据的目的。每个码字的长度近似与概率的负对数成比例。因此,最常见的符号使用最短的码。
(13)帧内(Intraframe)压缩,也称为空间压缩(Spatial compression)。当压缩一帧图像时,仅考虑本帧的数据而不考虑相邻帧之间的冗余信息,这实际上与静态图像压缩类 似。帧内压缩一般采用有损压缩算法。
(14)帧间(Interframe)压缩,是基于许多视频或动画的连续前后两帧具有很大的相关性,或者说前后两帧信息变化很小的特点。也即连续的视频其相邻帧之间具有冗余信息,根据这一特性,压缩相邻帧之间的冗余量就可以进一步提高压缩量,减小压缩比。帧间压缩也称为时间压缩(Temporal compression)。
(15)调制与编码策略(Modulation and Coding Scheme,MCS),802.11n射频速率的配置通过MCS索引值实现。MCS调制编码表是802.11n为表征WLAN的通讯速率而提出的一种表示形式。MCS将所关注的影响通讯速率的因素作为表的列,将MCS索引作为行,形成一张速率表。
(16)下采样,对于一个样值序列间隔几个样值取样一次,这样得到新序列就是原序列的下采样。采样率变化主要是由于信号处理的不同模块可能有不同的采样率要求。下采样相对于最初的连续时间信号而言,还是要满足采样定理才行,否则这样的下采样会引起信号成分混叠。下采样就是抽取,是多速率信号处理中的基本内容之一。
(17)图像编码,图像编码也称图像压缩,是指在满足一定质量(信噪比的要求或主观评价得分)的条件下,以较少比特数表示图像或图像中所包含信息的技术。
(18)服务质量(Quality of Service,QoS)是网络的一种安全机制,用来解决网络延迟和阻塞等问题的一种技术。当网络过载或拥塞时,QoS能确保重要业务量不受延迟或丢弃,同时保证网络的高效运行。
(19)YUV,是一种颜色编码方法。常使用在各个视频处理组件中。YUV在对照片或视频编码时,考虑到人类的感知能力,允许降低色度的带宽。YUV是编译true-color颜色空间(color space)的种类,Y'UV,YUV,YCbCr,YPbPr等专有名词都可以称为YUV,彼此有重叠。“Y”表示明亮度(Luminance、Luma),“U”和“V”则是色度、浓度(Chrominance、Chroma)
(20)信道容量(Channel capacity),又称为通道容量,是指在一个信道中能够可靠地传送信息时可达速率的最小上界。所谓可靠传输指的是可以以任意小的错误率传递信息。根据有噪信道编码定理,信道容量是可以误差概率任意小地达到的给定信道的极限信息率。信道容量的单位为比特每秒、奈特每秒等等
其次,分析并提出本申请的应用场景以及所具体要解决技术问题。
应用场景:在现有技术中,无线视频传输的发展,解决了不方便布线施工的的情况下音视频等数据进行传输的问题。因此,无线投屏将手机/平板电脑等终端的屏幕信息压缩,而后通过无线投送到电视等大屏幕设备上,通过无线网络向朋友分享视频/图片,通过无线网络进行视频会议等,都是图像编解码通过无线传输的典型应用场景之一。
常见的无线视频传输***一般由两个部分组成,分别是发射端、接收端。此外,可以根据实际情况的需求还可以添加适当的增益天线来加大传输距离。一般无线视频传输的发射端和接收端主要包括音视频编码、无线传输、音视频解码与显示等几个主要环节。在现有技术中,JPEG标准编码和MPEG标准编码是常见的两种图像编码方式,这两种图像编码方式都可以在原始图像经过色彩空间变换后转换为YUV空间,而后经过分割、离散余弦 变换(Discrete Cosine Transform,DCT)、量化、熵编码之后,得到压缩后图像,这两种编码方式在压缩倍数和图像质量方面都有很大的优势。但是,因为无线信道短时时变特性会导致信道容量急剧变化,在发射端的音视频编解码算法需要能快速跟踪并适应这一变化,令接收端即接收端的延迟和图像质量保持在可接受的水平,而如JPEG标准编码和MPEG标准编码这些仅仅着眼于压缩倍数和图像质量的传统音视频编码技术,并不能快速根据信道容量做出相应的编码调整。因此,很容易在无线传输过程中丢失部分甚至全部的码流信息。而且,在传统音视频编码技术所对应接收端只有具备某帧对应的全部码流才可以重构出完整的图像,少量的码流丢失即可导致马赛克、卡顿等现象,观感很差。此外,基于网络分层原理,无线传输时一般也不会关心信源部分(即,对音视频编码后的码流信息)的优先级,执行无差异的传输。因此,编码端并不能控制在无线信道中可能会丢失的是否是关键性信息(如:音频信息、重要帧的码流信息),而且无法快速适应无线信道的变化,也没有对信息源(如:码流信息或其他数据信息)进行分级发送,进而可能会由于重要信息的丢失导致接收端无法重构出完整的视频图像。
为了便于理解本申请实施例,下面先对本申请实施例所基于的其中一种低延迟信源信道联合编码***架构进行描述。请参阅附图1,图1是本申请实施例提供的一种低延迟信源信道联合编码***构架示意图。
本申请中的低延迟信源信道联合编码***构架可以包括图1中编码端10和解码端11。其中。低延迟信源信道联合编码***架构的编码端10包括视频源101、编码器102和发射机103。低延迟信源信道联合编码***架构的解码端11包括解码器111和显示设备112。其中,
视频源101是用于提供目标视频的接口、显存、存储器等,其中,目标视频可以来自各种类型接口,例如:高清多媒体(High Definition Multimedia Interface,HDMI)接口、数字式视频(Display Port,DP)接口、视频图形阵列标准(Video Graphics Array,VGA)接口等。例如:当视频源101为HDMI接口时,可以向编码器102发送目标视频,以便编码器102对目标视频的目标图像进行编码。
编码器102(encoder)是将信号(如:比特流)或数据进行编制、转换为可用以通讯、传输和存储的信号形式的设备。编码器能够把角位移或直线位移转换成电信号,前者称为码盘,后者称为码尺,编码器可以是相关编码软件运行在通用中央处理器(Central Processing Unit,CPU)或者专用芯片上、或者独立编码芯片、或者独立芯片的一部分,如:一种集成电路的***级(System-on-a-Chip,SoC)芯片的一部分。例如,当编码器为独立编码芯片时,可以对目标图像进行分层编码,即对目标图像下采样后获得基准帧图像和非基准帧图像进行图像编码;然后获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;最后将第一码流和第二码流通过发射机103发送给解码器111。其中,编码参数还可以接收通过编码参数反馈控制发送的根据当前无线信道的信道信息反馈的编码参数,用于使进行图像编码后的码流的码率小于当前无线信道的信道容量。例如,当编码器为独立编码芯片时,还可以对目标图像进行下采样,得到基准帧图像与非基准帧图像;根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数;根据编码参数, 对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。
发射机103是使用无线连接的局域网利用无线电波作为数据传送的设备,传送距离一般为几十米。发射机103可以是WiFi芯片,可能包括与WiFi芯片配套的驱动程序、固件等等。例如:当发射机103为WiFi芯片时,可以将编码器102进行图像编码后获得第一码流进而第二码流分别通过无线信道发送,其中,通过无线发送第一码流时的信道资源高于发送第二码流时的信道资源。需要说明的是,发射机103还可以获得当前无线信道的信道环境,如:信道容量、信道的带宽或信号与干扰加噪声比等;然后通过编码参数反馈控制向编码器103发送信道信息,以便编码器103实时调整第一码流和第二码流的编码参数;发射机103也可以实时调整第一码流和第二码流的发送参数。其中,编码参数反馈控制在编码器内部或外部可以作为一种算法,可能是软件实现,也可以是芯片实现,编码参数反馈控制主要是用于根据当前无线信道的信道信息调整编码器在进行图像编码时的编码参数。可以理解的是,编码器102和发射机103可以是耦合在一起的装置,用于接收来自视频源101发送的目标视频信息,并将该目标视频信息进行图像编码后发送给解码器111。
可选的,解码器(decoder)111,是一种能将数字视音频数据流解码还原成模拟视音频信号的硬件/软件设备。编码器主要把模拟视音频信号压缩数据编码文件,而解码器把数据编码文件转为模拟视音频信号的过程。解码器可以是相关解码软件运行在通用CPU或者专用芯片上、或者独立解码芯片、或者独立芯片(如SoC芯片)的一部分;例如:当解码器111为独立解码芯片时,解码器111可以接收接收编码端的发射机103发送的第一码流,第一码流为基准帧图像进行图像编码后获得的码流,其中,基准帧图像包括对目标图像进行下采样后获得的基准帧图像;然后解码器111按照与编码器102对目标视频的目标图像进行编码时相反的流程解码第一码流得到基准帧;最后根据该基准帧获得目标图像。又例如:解码器111还可以接受第二码流,再按照与编码器102对目标视频的目标图像进行编码时相反的流程解码第二码流得到完整或残缺的非基准帧;最后解码器利用基准帧和完整非基准帧拼接出完整的重构帧,即目标图像;当非基准帧残缺时,可以利用相邻帧图像(如:相邻一帧目标图像的基准帧图像和非基准帧图像中的至少一个)或周边像素集合插值得到重构帧,而后将获得的重构帧发送到显示设备112中显示并播放音频。还可以理解的是,解码器111在确定非基准帧是否残缺时,还可以在接收到码流时就确定该码流是否残缺进而确定该码流对应的图像是否残缺,本申请对此不作限定。
可选的,显示设备112,是一种将一定的数据通过特定的传输设备显示到屏幕上再反射到人眼的显示工具,一般可以指显示器、投影仪、虚拟现实头戴式显示设备、带有显示功能的智能终端等设备。例如,显示设备112可以是:虚拟现实头戴式显示设备(如:虚拟现实(Virtual Reality,VR)眼镜、VR眼罩、VR头盔)、智能手机、笔记本电脑、平板设备、投影仪、摄影机等,或安装于或运行于电脑、平板设备、智能手机上的显示客户端、应用程序等。例如:当显示设备为投影仪时,该投影仪可以在接收到解码器111发送的目标视频的数据信息后,通过投影设备投射到屏幕上,进而播放目标视频。可以理解的是,解码器111和显示设备112可以是耦合在一起的装置,用于接收来自发射机103发送的码流信息,将该码流信息解码后显示该码流信息对应的目标视频。
需要说明的是,本申请实施例提供的一种低延迟信源信道联合编码***架构中的解码端11相当于本申请中的接收端。
还需要说明的是,图1的低延迟信源信道联合编码***架构只是本申请实施例中的部分示例性的实施方式,本申请实施例中的低延迟信源信道联合编码***架构包括但不仅限于以上低延迟信源信道联合编码***架构。
基于上述技术问题分析和上述的低延迟信源信道联合编码***架构,本申请提出一种低延迟信源信道联合编码方法,能够避免传统音视频编解码算法中对码流完整性的要求,也能够快速跟踪信道容量的变化,使接收端延迟和图像质量保持在可接受的水平。在信道容量降低时,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
请参见附图2A,图2A是本申请实施例提供的一种低延迟信源信道联合编码方法的流程示意图,该方法可应用于上述图1中所述的低延迟信源信道联合编码***架构中,其中的编码器102和发射机103可以用于支持并执行图2A中所示的方法流程步骤S201-步骤S203。
步骤S201:对目标图像进行下采样,得到基准帧图像与非基准帧图像。
具体地,编码器102对目标图像进行下采样,得到基准帧图像与非基准帧图像,下采样,是对于一个样值序列间隔几个样值取样一次,这样得到新序列就是原序列的下采样。请参考附图2B,图2B是本申请提供的一种应用场景示意图。例如:一段3秒钟的目标视频在需要进行无线传输时,编码器102可以对输入的目标视频的多帧目标图像进行下采样,其中,下采样的水平采样率为2:1,竖直采样率为2:1,分割为4个较低分辨率的副本图像,其中,可以选择一个副本图像归为基准帧图像,剩余3个副本图像为非基准帧图像。可以理解的是,基准帧图像可以包括一个或多个图像,非基准帧图像也可以包括一个或多个图像,而基准帧图像包括的图像数量要小于或等于非基准帧图像包括的图像数量。在对目标图像进行下采样时,可以选择是目标图像的一半分辨率进行采样,也可以是目标图像的任意比例的分辨率进行采样。下采样过后的副本图像的分辨率要小于目标图像的分辨率,因此,后续对副本图像压缩后获得的码流的码率大小要小于目标图像在使用相同的编码技术压缩时获得的码流的码率大小,进而可以在信道容量发生抖动,或者降低至初始码率(例如:初始码率可以为按照传统编码方法对目标图像进行压缩时的码率大小或预设大小的码率)以下时,使得接收端的解码器仍然可以基于较小分辨率副本图像得到的重构图像。可选的,目标视频包括多帧目标图像,目标图像为多帧目标图像中的任意一个。
可选的,编码器102对目标图像进行下采样获得N个副本图像,N个副本图像的分辨率低于目标图像的分辨率,其中,N为大于1的正整数;将上述N个副本图像划分为基准帧图像与非基准帧图像。需要说明的是,编码器102对目标图像进行下采样,获得多个分辨率较低的副本图像后,可以将该多个分辨率较低的副本图像分为数量不等的两种图像,其中,基准帧图像包含的图像数目小于或等于非基准帧图像包括的图像数目。还可以理解的是,所述基准帧图像与非基准帧图像的交集为空集。如图2B所示,在下采样后获得的4个副本图像,其中,可以选择任意1个副本图像为基准帧图像,剩余3个副本图像为非基准帧图像。
步骤S202:对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。
具体地,编码器102对多个副本图像划分后的基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。例如:编码器102可以对基准帧图像中的副本图像依次进行分块、DCT、量化、游长编码、熵编码、打包、标记等等生成第一码流。编码器102可以对非基准帧图像中的每个副本图像相对与基准帧图像中的副本图像的残差或者直接对非基准帧图像中的每个副本图像,进行分块、DCT、量化、游长编码、熵编码、打包、标记等等生成第二码流。
可选的,编码器102可以对基准帧图像和非基准帧图像分别进行帧内压缩,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。可以理解的是,在选择较为简便的下采样将目标图像分层,降低分辨率的同时,对分层后的副本图像进行图像编码时使用的帧内压缩编码方法,由于帧内压缩是空域压缩,当压缩一帧图像时,仅考虑本帧的数据而不考虑相邻帧之间的冗余信息,而帧间压缩要参考其他帧数据,压缩率大,考虑到图像延迟方面,本申请实施例对下采样后的副本图像进行图像编码时并不适用帧间压缩。因此,使用帧内压缩的方法可以在信道容量发生抖动,并降低至初始码率以下时,可以使得解码器能够接受到基于较小分辨率副本图像图像编码后的码流信息,能够根据该码流信息得到的重构图像。同时在信道容量降低时,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
可选的,编码器102还可以用其他分层编码的方法,将目标图像也分为较低分辨率的副本图像,该多个副本图像之间的分辨率、码率、帧率都可以不同,也可以相同。例如:分层视频编码产生的码流中包含多个子码流,子码流分为基本层和扩展层,各层具有不同的码率,帧率和分辨率,基本层拥有最基本的视频质量,后面每一个扩展层都是对上一层的补充,拉流端可以根据实际的网络环境(如:信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率等)来选择解码几路子码流。
可选的,编码器102在对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流之前,还可以基于当前无线信道的信道环境,确定所述基准帧图像和所述非基准帧图像对应的编码参数,其中,所述信道环境可以包括信道容量,所述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流,所述第一码流和所述第二码流分别对应的所述目标码率均小于或等于所述信道容量。需要说明的是,该编码器确定的编码参数可以是编码参数反馈控制发送的编码参数。因此,可以在当信道容量发生变化时,通过发射机获取当前信道的信道容量,根据该反馈回的信道容量信息调整编码器在对副本图像进行图像编码时的编码参数,使得编码后的码流的码率可以在适应于当前的信道容量的情况下,能够顺利发送目标码流,减少因为信道容量降低,造成码流丢失的情况。同时也降低因无法传输完整码流,使得接收端的解码器不能够根据该缺失码流信息得到的重构图像,降低了视频图像卡顿、模糊的概率,提升了用户的观看体验。
步骤S203:基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流。
具体地,发射机103可以将从编码器102接收到的第一码流与第二码流,在基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源后,分别利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流。如图2B所示,本申请实施例可以基于反馈的当前无线信道的信道环境,分别发送第一码流和第二码流。例如:可以设置第一码流的优先级高于第二码流的优先级,然后在分别发送第一码流与第二码流时,根据当前无线信道的信道环境,使得在发送第一码流时信道的服务质量高于发送第二码流时信道的服务质量。或者,当前无线信道的信道环境较差时,可以优先发送比较重要的第一码流,再发送级别比较低的第二码流;或者,当前无线信道的信道环境较差时,在接收端接收到第一码流后或者在发送第一码流后的一段时间后,再发送级别比较低的第二码流。
可选的,发射机103可以将SINR/SNR/RSSI等链路统计信息经空口发送给编码端,编码端统计误子帧率、碰撞率等,然后再根据链路统计信息和误子帧率、碰撞率等调整发送参数,如:发送MCS、发射功率、量化参数等。即,当信道发生变化时,通过发射机获取当前信道的信道信息(如:信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率等),根据该信道信息调整在发送第一码流和/或发送第二码流时的发送参数(如:发射功率、调制与编码策略信息等),例如:发射机103可以基于当前的所述信道环境,确定所述第一码流和/或所述第二码流分别对应的发送参数,其中,所述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,所述第一码流的发送参数优于所述第二码流的发送参数;按照所述第一码流对应的发送参数发送所述第一码流,以及按照所述第二码流对应的发送参数发送所述第二码流。其中,所述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个。可以理解的是,发射机103在发送第一码流和第二码流时,会更快的响应第一码流的发送请求。在无线传输的过程中,将视频源的重要性和优先性分级,即,发送第一码流时的传输质量高于发送第二码流时的传输质量,或者,优先发送比较重要码流(即,副本图像数量少进行图像编码后的码流),使得在信道容量等发生抖动时,解码器无法接收到完整的码流时,可以仅仅基于部分码流对应的副本图像得到的重构图像,可以降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
可选的,发射机103还可以基于当前的信道环境,确定所述第一无线信道和所述第二无线信道,将所述第一码流通过第一无线信道发送,将所述第二码流通过第二无线信道发送,其中,所述第一无线信道的服务质量保障机制高于所述第二无线信道。其中,所述无线信道包括第一无线信道和第二无线信道。请参考附图2C,图2C是本申请实施例提供的一种第一码流和第二码流分别通过第一信道和第二信道发送的示意图。发射机103将第一码流通过第一无线信道发送,将第二码流通过第二无线信道发送。在信道容量等发生抖动时,通过无线信道发送的第一码流和第二码流分别使用不同的QoS保障级别,解码器可以仅仅接收到通过服务质量保障机制高的信道发送的第一码流,然后可以仅仅通过第一码流重构图像,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。可以理解的是,根据信道信息调整发送参数后,第一码流的发送参数与第二码流的发送参数不同,第一码流的发送参数与第二码流的发送参数相比能够使发射机以更迅速的发送速度、更高的QoS发送第一码流。即,以更好地信道资源,更高的传输成功率发送第一码流,该成功 率可以指码流信息被完整传输到接收端的概率。因此,在无线传输的过程中,可以将视频源分级,优先发送比较重要码流,使得在信道容量等发生抖动时,或者解码器无法接收到完整的码流时,可以仅仅基于部分码流对应的副本图像得到的重构图像,例如:将第一码流与第二码流经无线信道发送,将第一码流映射到更高服务质量(Quality of Service,QoS)保障机制,如:将第一码流映射到VI业务,将第二码流映射到BE业务。在无线传输的过程中,还可以通过同一个无线通道发送第一码流和第二码流,但是可以优先只发送第一码流,保证接收端至少可以接收到第一码流。又例如:发送第一码流与第二码流时,可以先发送第一码流,只有在在接收端接收到第一码流后或者在发送第一码流后的预设时间段之后,才发送第二码流,以确保接收端至少能获得一个完整的、较低分辨率的码流信息,以便重构图像。将第一码流与第二码流分别通过无线信道发送,可以使得接收端即使在无法接收到完整的码流时,提高接收端接收到第一码流的概率,使得接收端在仅仅接收到的第一码流时还可以得到完整的重构图像。
可选的,获取所述目标视频包括的多帧目标图像中每一帧目标图像对应所述基准帧图像的码流;获取所述目标视频中音频信息并标记为音频信息;将所述目标视频包括的多帧图像中每一帧图像对应所述基准帧图像的码流和所述音频信息通过所述第一无线信道发送。可以理解的,一个视频中,音频信息与多帧目标图像的第一码流信息同等重要,因此为避免音频信息的丢失,可以在发送第一码流时可以与音频信息一起经过QoS保障级别较高的第一信道发送,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。
可选的,获取所述目标视频包括的多帧目标图像中每一帧目标图像对应所述非基准帧图像的码流;将所述目标视频包括的多帧图像中每一帧图像对应所述非基准帧图像的码流通过所述第二无线信道发送。在获取目标视频包括的多帧目标图像中所有目标图像对应上述非基准帧图像的码流后,单独映射到服务质量保障机制稍低的BE业务发送。因此,即使所有目标图像的非基准帧图像对应的码流在信道发生变化时丢失部分甚至是全部码流信息时,解码器可以仅仅基于通过服务质量保障机制高的信道接收到的所有目标图像的第一副本对应的码流和音频信息重构视频,降低视频信息出现马赛克,卡顿、模糊等现象,提升用户的观感体验。
在本申请实施例中,可以对目标图像进行下采样,获得基准帧图像与非基准帧图像,然后对基准帧图像进行图像编码获得第一码流,对非基准帧图像进行图像编码获得第二码流,最后利用第一信道资源发送第一码流,以及利用第二信道资源发送第二码流。其中,第一信道资源以及第二信道资源根据当前无线信道的信道环境确定,且第一信道资源优于第二信道资源,即可以理解为,在无线传输的过程中,可以将基准帧图像编码后的第一码流与非基准帧图像编码后的第二码流分级发送,例如:可以设置第一码流的优先级高于第二码流的优先级,然后在分别发送第一码流与第二码流时,根据当前无线信道的信道环境,使得在发送第一码流时信道的服务质量高于发送第二码流时信道的服务质量。或者,当前无线信道的信道环境较差时,可以优先发送比较重要的第一码流,再发送级别比较低的第二码流;或者,当前无线信道的信道环境较差时,在接收端接收到第一码流后或者在发送第一码流后的一段时间后,再发送级别比较低的第二码流。基于当前无线信道的信道环境,分别发送第一码流和第二码流,可以使得接收端在无法接收到完整的码流时,提高接收端 接收到第一码流的概率,利用接收到的第一码流得到完整的重构图像。其次,在对目标图像进行图像编码前,先对目标图像进行下采样,在获得多个分辨率较低的副本图像后,再进行图像编码,可以使得进行下采样后的获得码流要小于直接对目标图像进行图像编码时获得的码流大小,进而更容易使得接收端的解码器接收到完整的码流。因此,对目标图像通过下采样进行图像编码后,将编码后的码流信息基于当前无线信道的信道环境,分别通过无线信道发送,可以使得编码后的码流信息仍可以适应实时变化的信道环境,进而降低在无线传输过程中信息丢失的概率,进而避免由于信息丢失导致接收端无法重构完整的视频图像,提升用户的观看体验。
参见图3A和图3B,图3A是本申请实施例提供的一种低延迟信源信道联合编解码方法的流程示意图,图3B是本申请实施例提供的一种对目标图像进行编解码应用示意图。图3A所示方法可应用于上述图1中所述的低延迟信源信道联合编码***架构中,其中的编码器102和发射机103可以用于支持并执行图3A中所示的方法流程步骤S301-步骤S304,解码器111可以用于支持并执行图3A中所示的方法流程步骤S305-步骤S307。该方法可以包括以下步骤S301-步骤S307。
步骤S301:对目标图像进行下采样,得到基准帧图像与非基准帧图像。
具体地,上述对步骤S301的描述可以对应参照图2A的步骤S201的相关描述,此处不再赘述。
步骤S302:根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数。
具体的,编码器102根据当前无线信道的信道环境,确定所述基准帧图像和所述非基准帧图像对应的编码参数,请参考附图3C,图3C是本申请提供的另一种应用场景示意图,即可以根据当前无线信道的信道环境,确定图形编码时的编码参数。例如:一段3秒钟的目标视频在需要进行无线传输时,编码器102可以对输入的目标视频的多帧目标图像进行下采样,其中,下采样的水平采样率为2:1,竖直采样率为2:1,分割为4个较低分辨率的副本图像,其中,选择一个副本图像归为基准帧图像,剩余3个副本图像为非基准帧图像,然后基于当前无线信道的信道环境,确定所述基准帧图像和所述非基准帧图像在图像编码时对应的编码参数,以减小编码后的码流大小。
可选的,发射机103可以获取当前无线信道的信道环境,其中,信道环境包括信道容量。在当信道容量发生变化时,通过发射机获取当前信道的信道容量,可以根据该信道容量调整编码器在对副本图像进行图像编码时的编码参数,使得编码后的码流的码率在适应于当前的信道容量的情况下,能够顺利发送目标码流,减少因为信道容量降低,而造成码流丢失的情况。
步骤S303:根据编码参数,对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。
具体地,编码器102通过编码参数反馈控制根据信道容量调整在进行图像编码时的基准帧图像和非基准帧图像分别对应的编码参数,然后根据调整后确定的编码参数,对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码 流,和所述非基准帧图像编码后的第二码流。其中,编码参数用于控制对应的副本在进行图像编码时按照目标码率生成对应的码流,以便第一码流和第二码流的码率均小于或等于上述信道容量,保证编码后码流的码率适应于当前的信道容量。因此,根据当前无线信道的信道信息,调整图像编码时的编码参数,可以令生成的码流的码率低于当前无线信道的信道容量,进而码流在无线传输时,不会因为信道容量低至无法传输完整码流,或者信道容量发生抖动,又或者信道容量降低至初始码率(如:初始码率可以为按照传统编码方法对目标图像进行压缩时的码率或者预设码率)以下时,而丢失部分码流信息。因此,本申请实施例的图像编解码算法能快速跟踪并适应无线信道的信道环境的变化,令接收端的延迟和图像质量保持在可接受的水平。
可选的,第一码流和第二码流的码率可以相同也可以不同,当第一码流和第二码流的码率相同时可以统一调控图像编码时的编码参数,当第一码流和第二码流的码率不同时,首先要求保证第一码流的码率小于或等于信道容量,减小由于信道容量降低而造成码流丢失的概率。
步骤S304:分别发送第一码流以及第二码流。
具体地,上述对步骤S304的描述可以对应参照图2A的步骤S203的相关描述,此处不再赘述。
步骤S305:接收编码端发送的第一码流。
具体地,解码器111接收编码端102发送的第一码流,所述第一码流为基准帧图像进行图像编码后获得的码流。本申请实施例可以在无线传输的过程中,接收编码端发送的第一码流,进而将第一码流进行图像解码后,获得第一码流对应的基准帧图像,然后根据基准帧图像,获得目标图像。需要说明的是,第一码流为基准帧图像进行图像编码后获得的码流,其中,基准帧图像包括一个或多个对目标图像进行下采样后获得的图像,因此,编码端接收的码流是对目标图像进行下采样再图像编码获得的。下采样可以将目标图像分为多个分辨率较低的副本图像,选择其中一部分作为基准帧图像。而且对目标图像进行下采样再编码获得的码流大小要小于直接对目标图像进行图像编码时获得的码流大小,进而下采样后的码流可能会更好适应无线信道的变化,并在传输过程中减少了信息丢失的概率,同时也避免了由于信息丢失导致接收端无法重构完整的视频图像
步骤S306:对第一码流进行图像解码后,获得第一码流对应的基准帧图像。
具体地,解码器111将所述第一码流进行图像解码后,获得所述第一码流对应的所述基准帧图像。编码器102将基准帧图像进行分块、DCT、量化、游长编码、熵编码、打包、标记,即获得对应的第一码流;则解码器111可以按照与编码器102完全相反的过程即,解标记,解包,反熵编码,反游长编码,反量化,反DCT,解码第一码流得到YUV,再由YUV转化成RGB得到基准帧图像。如图3B所示编码端的编码器将目标图像由RGB转化成YUV,再依次进行分块、DCT、量化、游长编码、熵编码、打包、标记;然后通过发送到接收端,接收端的解码器按照与编码相反的流程解码第一码流得到YUV,再由YUV转化成RGB得到基准帧。
步骤S307:根据基准帧图像,重构目标图像。
具体地,解码器111将根据所述基准帧图像,重构所述目标图像。其中,所述目标图 像为目标视频包括的多帧目标图像中的任意一个;所述根据所述基准帧图像,重构所述目标图像,包括:根据所述基准帧图像与所述目标图像相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,通过插值算法重构所述目标图像。例如:当在进行图像编码时可以对码流标记一个时间包;当接收到第一码流后,在所述时间包规定的时间段内还没有接收到第二码流,即可认为第二码流丢失,此时可以根据接收到的基准帧码流和前一帧插值得到重构帧,即,目标图像。因此,当解码端接收到第一码流时,即可以避免由于信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。
可选的,解码器111在接收所述第一码流后的预设时间段内,接收所述编码端发送的第二码流,其中,所述第二码流为非基准帧图像进行所述图像编码后获得的码流,所述非基准帧图像包括对所述目标图像进行下采样后获得的除所述基准帧图像外剩余的图像;若所述第二码流残缺,将所述第二码流进行图像解码后,获得对应的残缺非基准帧图像;根据所述残缺非基准帧图像,确定所述残缺非基准帧图像的周边像素;所述根据所述基准帧图像,重构所述目标图像,包括:根据所述基准帧图像与所述残缺非基准帧图像的周边像素,通过插值算法重构所述目标图像。可以理解的,在接受到第一码流的预设时间段内,确定是否接收到第二码流,若否,则根据第一码流对应的基准帧图像与目标图像的所述目标图像相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,重构目标图像;若接收到了第二码流,则可以确定接收到的上述第二码流是否残缺,若接收到的第二码流残缺则可以利用残缺的第二码流与完整的第一码流插值得到目标图像,避免了由于部分信息丢失导致接收端无法重构完整的视频图像,降低视频信息出现无声音、卡顿等现象,提升用户的观感体验。如:当在进行图像编码时可以对码流标记一个时间包;当接收到第一码流后,在所述时间包规定的时间段内接收了第二码流,此时需要确定第二码流是否残缺,若确定所述第二码流残缺,则可以根据残缺的第二码流信息利用插值算法与完整的第一码流信息重构目标图像;或者可以根据残缺的第二码流信息利用插值算法与完整的第一码流信息以及目标图像的相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个重构目标图像。
可选的,解码器111在接收所述第一码流后的预设时间段内,接收所述编码端发送的第二码流;若所述第二码流完整,将所述第二码流进行图像解码后,获得对应的所述非基准帧图像;所述根据所述基准帧图像,重构所述目标图像,包括:将所述基准帧图像和所述非基准帧图像拼接,以重构所述目标图像。例如:当在进行图像编码时可以对码流标记一个时间包;当接收到第一码流后,在所述时间包规定的时间段内接收了第二码流,若确定所述第二码流完整,则可以直接将第一码流解码后的基准帧图像与第二码流解码后的非基准帧图像拼接成目标图像。
实施本申请实施例时,本申请实施例的图像编解码算法能快速跟踪并适应无线信道的信道环境的变化,令接收端的延迟和图像质量保持在可接受的水平。首先,编码端根据当前无线信道的信道信息,调整图像编码时的编码参数,可以令生成的码流的码率低于当前无线信道的信道容量,进而码流在无线传输时,不会因为信道容量低至无法传输完整码流,或者信道容量发生抖动,又或者信道容量降低至初始码率(如:初始码率可以为按照传统 编码方法对目标图像进行压缩时的码率或者预设码率)以下时,而丢失部分码流信息。其次,编码端接收的码流是对目标图像进行下采样再图像编码获得的。又由于下采样可以将目标图像分为多个分辨率较低的副本图像,则对目标图像进行下采样再编码获得的码流大小要小于直接对目标图像进行图像编码时获得的码流大小,进而下采样后的码流可能会更好适应无线信道的变化,并在传输过程中减少了信息丢失的概率,同时也避免了由于信息丢失导致接收端无法重构完整的视频图像。同时发送参数的调整,信道资源的分配,也降低了在传输过程中信息丢失的概率,避免由于信息丢失导致接收端无法重构完整的视频图像,降低了视频信息出现马赛克,卡顿、模糊等现象,提升用户的观看体验。
上述详细阐述了本申请实施例的方法,下面提供了本申请实施例的相关装置。
请参见附图4A,图4A是本申请实施例提供的一种低延迟信源信道联合编码装置的结构示意图,该低延迟信源信道联合编码装置10可以包括第一采样单元401、第一编码单元402和第一发送单元403,还可以包括:第一编码参数单元404、第一获取单元405、第二发送单元406、第二获取单元407和第三发送单元408其中,各个单元的详细描述如下。
第一采样单元401,用于对目标图像进行下采样,得到基准帧图像与非基准帧图像。
第一编码单元402,用于对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。
第一发送单元403,用于基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流,其中,第一信道资源优于第二信道资源。
在一种可能实现的方式中,上述第一编码单元402,具体用于将上述基准帧图像包括的每个上述基准帧图像进行帧内压缩获得上述第一码流;将上述非基准帧图像与上述基准帧图像的残差进行上述帧内压缩获得对应的上述第二码流。
在一种可能实现的方式中,上述信道环境还包括信道容量;所述装置还包括:第一编码参数单元404,用于在上述对上述基准帧图像和上述非基准帧图像分别进行图像编码,获得上述基准帧图像编码后的第一码流,和上述非基准帧图像编码后的第二码流之前,基于上述信道容量,确定上述基准帧图像和上述非基准帧图像对应的编码参数,其中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流,上述第一码流和上述第二码流分别对应的上述目标码率均小于或等于上述信道容量。
在一种可能实现的方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;第一发送单元403,具体用于基于当前的上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数;按照上述第一码流对应的发送参数发送上述第一码流,以及按照上述第二码流对应的发送参数发送上述第二码流。
在一种可能实现的方式中,上述无线信道包括第一无线信道和第二无线信道;第一发送单元403,具体用于:基于当前的信道环境,确定上述第一无线信道和上述第二无线信道,将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其 中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第一获取单元405,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;第二发送单元406,用于将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。
在一种可能实现的方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第二获取单元407,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;第三发送单元408,用于将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。
需要说明的是,本申请实施例中所描述的低延迟信源信道联合编码装置20中各功能单元的功能可参见上述图2A-图2C中所述的方法实施例中的相关描述,此处不再赘述。
请参见附图4B,图4B是本申请实施例提供的另一种低延迟信源信道联合编码装置的结构示意图,该低延迟信源信道联合编码装置20可以包括第二采样单元411、第二编码参数单元412、第二编码单元413和第四发送单元414,还可以包括:第三获取单元415、第五发送单元416、第四获取单元417和第六发送单元418其中,各个单元的详细描述如下。
第二采样单元411,用于对目标图像进行下采样,得到基准帧图像与非基准帧图像。
第二编码参数单元412,用于根据当前无线信道的信道环境,确定基准帧图像和非基准帧图像对应的编码参数。
第二编码单元413,用于根据编码参数,对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流。
第四发送单元414,用于分别发送第一码流以及第二码流。
在一种可能的实现方式中,上述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流;上述第二编码单元413,具体用于:将上述基准帧图像包括的每个上述基准帧图像按照上述目标码率进行帧内压缩,获得上述第一码流;将上述非基准帧图像与上述基准帧图像的残差按照上述目标码率进行上述帧内压缩,获得对应的上述第二码流。
在一种可能的实现方式中,上述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;上述第四发送单元414,用于分别发送上述第一码流以及上述第二码流之前,还用于:根据上述信道环境,确定上述第一码流和/或上述第二码流分别对应的发送参数,其中,上述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,上述第一码流的发送参数优于上述第二码流的发送参数。
在一种可能的实现方式中,上述无线信道包括第一无线信道和第二无线信道;上述第四发送单元414,具体用于:将上述第一码流通过第一无线信道发送,将上述第二码流通过第二无线信道发送,其中,上述第一无线信道的服务质量保障机制高于上述第二无线信道。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第三获取单元415,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述基准帧图像的码流;获取上述目标视频中音频信息;第五发送单元416,用于将上述目标视频包括的多帧图像中每一帧图像对应上述基准帧图像的码流和上述音频信息通过上述第一无线信道发送。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述装置还包括:第四获取单元417,用于获取上述目标视频包括的多帧目标图像中每一帧目标图像对应上述非基准帧图像的码流;第六发送单元418,用于将上述目标视频包括的多帧图像中每一帧图像对应上述非基准帧图像的码流通过上述第二无线信道发送。
需要说明的是,本申请实施例中所描述的低延迟信源信道联合编码装置20中各功能单元的功能可参见上述图3A-图3C中所述的方法实施例中的相关描述,此处不再赘述。
请参见附图4C,图4C是本申请实施例提供的一种低延迟信源信道联合解码装置的结构示意图,该低延迟信源信道联合解码装置30可以包括接收单元421、解码单元422和图像单元423,还可以包括:第五获取单元424和第六获取单元425,其中,各个单元的详细描述如下。
接收单元421,用于接收编码端发送的第一码流,上述第一码流为基准帧图像进行图像编码后获得的码流,其中,上述基准帧图像包括对目标图像进行下采样后获得的基准帧图像。
解码单元422,用于将上述第一码流进行图像解码后,获得上述第一码流对应的上述基准帧图像。
图像单元423,用于根据上述基准帧图像,重构上述目标图像。
在一种可能的实现方式中,上述目标图像为目标视频包括的多帧目标图像中的任意一个;上述图像单元423,具体用于:根据所述基准帧图像与所述目标图像相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,通过插值算法重构所述目标图像。
在一种可能的实现方式中,所述装置还包括:第五获取单元424,用于在接收上述第一码流后的预设时间段内,接收上述编码端发送的第二码流,其中,上述第二码流为非基准帧图像进行上述图像编码后获得的码流,上述非基准帧图像包括对上述目标图像进行下采样后获得的除上述基准帧图像外剩余的图像;若上述第二码流残缺,将上述第二码流进行图像解码后,获得对应的残缺非基准帧图像;上述图像单元423,具体用于:根据上述基准帧图像与上述残缺非基准帧图像的周边像素,通过插值算法重构上述目标图像。
在一种可能的实现方式中,所述装置还包括:第六获取单元425,用于若上述第二码流完整,将上述第二码流进行图像解码后,获得对应的上述非基准帧图像;上述图像单元423,具体用于:将上述基准帧图像和上述非基准帧图像拼接,以重构上述目标图像。
需要说明的是,本申请实施例中所描述的低延迟信源信道联合编码装置20中各功能单元的功能可参见上述图3A-图3C中所述的方法实施例中的相关描述,此处不再赘述。
如图5所示,图5是本申请实施例提供的一种终端设备的结构示意图,该终端设备40 是可以进行图像无线传输的智能终端,如投影仪、录像机、手机、平板电脑,车载终端等等,该设备包括至少一个处理器501,至少一个存储器502、至少一个通信接口503。此外,该设备还可以包括天线等通用部件,在此不再详述。
处理器501可以是通用中央处理器(CPU),微处理器,特定应用集成电路(application-specific integrated circuit,ASIC),现场可编程门阵列(Field Programmable Gata Array,FPGA)电路,或一个或多个用于控制以上方案程序执行的集成电路。
通信接口503,用于与其他设备或通信网络通信,如以太网,无线接入网(RAN),核心网,无线局域网(Wireless Local Area Networks,WLAN)等。
存储器502可以是只读存储器(read-only memory,ROM)或可存储静态信息和指令的其他类型的静态存储设备,随机存取存储器(random access memory,RAM)或者可存储信息和指令的其他类型的动态存储设备,也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory,EEPROM)、只读光盘(Compact Disc Read-Only Memory,CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。存储器可以是独立存在,通过总线与处理器相连接。存储器也可以和处理器集成在一起。
其中,所述存储器502用于存储执行以上方案的应用程序代码,并由处理器501来控制执行。所述处理器501用于执行所述存储器502中存储的应用程序代码。
存储器502存储的代码可执行以上图2A提供的低延迟信源信道联合编码方法,比如:对目标图像进行下采样,得到基准帧图像与非基准帧图像;对基准帧图像和非基准帧图像分别进行图像编码,获得基准帧图像编码后的第一码流,和非基准帧图像编码后的第二码流;基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用第一信道资源发送第一码流以及利用第二信道资源发送第二码流,其中,第一信道资源优于第二信道资源。
存储器502存储的代码还可执行以上图3A提供的低延迟信源信道联合编码方法,比如:对目标图像进行下采样,得到基准帧图像与非基准帧图像;根据当前无线信道的信道环境,确定所述基准帧图像和所述非基准帧图像对应的编码参数;根据所述编码参数,对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流;分别发送所述第一码流以及所述第二码流。
需要说明的是,本申请实施例中所描述的低延迟信源信道联合编码装置30中各功能单元的功能可参见上述图2A-图3C中所述的方法实施例中的相关描述,此处不再赘述。
在上述实施例中,对各个实施例的描述都各有侧重,某个实施例中没有详述的部分,可以参见其他实施例的相关描述。
需要说明的是,对于前述的各方法实施例,为了简单描述,故将其都表述为一系列的动作组合,但是本领域技术人员应该知悉,本申请并不受所描述的动作顺序的限制,因为依据本申请,某些步骤可能可以采用其他顺序或者同时进行。其次,本领域技术人员也应 该知悉,说明书中所描述的实施例均属于优选实施例,所涉及的动作和模块并不一定是本申请所必须的。
在本申请所提供的几个实施例中,应该理解到,所揭露的装置,可通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如上述单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,装置或单元的间接耦合或通信连接,可以是电性或其它的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外,在本申请各实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
上述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以为个人计算机、服务端或者网络设备等,具体可以是计算机设备中的处理器)执行本申请各个实施例上述方法的全部或部分步骤。其中,而前述的存储介质可包括:U盘、移动硬盘、磁碟、光盘、只读存储器(Read-Only Memory,缩写:ROM)或者随机存取存储器(Random Access Memory,缩写:RAM)等各种可以存储程序代码的介质。
以上所述,以上实施例仅用以说明本申请的技术方案,而非对其限制;尽管参照前述实施例对本申请进行了详细的说明,本领域的普通技术人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。
Claims (22)
- 一种低延迟信源信道联合编码方法,其特征在于,包括:对目标图像进行下采样,得到基准帧图像与非基准帧图像;对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流;基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用所述第一信道资源发送所述第一码流以及利用所述第二信道资源发送所述第二码流,其中,所述第一信道资源优于所述第二信道资源。
- 根据权利要求1所述方法,其特征在于,所述对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流,包括:将所述基准帧图像包括的每个所述基准帧图像进行帧内压缩获得所述第一码流;将所述非基准帧图像与所述基准帧图像的残差进行所述帧内压缩获得对应的所述第二码流。
- 根据权利要求1或2所述方法,其特征在于,所述信道环境还包括信道容量;所述对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流之前,还包括:基于所述信道容量,确定所述基准帧图像和所述非基准帧图像对应的编码参数,其中,所述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流,所述第一码流和所述第二码流分别对应的所述目标码率均小于或等于所述信道容量。
- 根据权利要求3所述方法,其特征在于,所述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;所述基于当前的信道环境,确定第一信道资源以及第二信道资源,分别利用所述第一信道资源发送所述第一码流以及利用所述第二信道资源发送所述第二码流,包括:基于当前的所述信道环境,确定所述第一码流和/或所述第二码流分别对应的发送参数,其中,所述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,所述第一码流的发送参数优于所述第二码流的发送参数;按照所述第一码流对应的发送参数发送所述第一码流,以及按照所述第二码流对应的发送参数发送所述第二码流。
- 根据权利要求1-4所述任意一项方法,其特征在于,所述无线信道包括第一无线信道和第二无线信道;所述基于当前的信道环境,确定第一信道资源以及第二信道资源,分别利用所述第一信道资源发送所述第一码流以及利用所述第二信道资源发送所述第二码流,包括:基于当前的信道环境,确定所述第一无线信道和所述第二无线信道,将所述第一码流通过第一无线信道发送,将所述第二码流通过第二无线信道发送,其中,所述第一无线信道的服务质量保障机制高于所述第二无线信道。
- 根据权利要求5所述方法,其特征在于,所述目标图像为目标视频包括的多帧目标图像中的任意一个;所述方法还包括:获取所述目标视频包括的多帧目标图像中每一帧目标图像对应所述基准帧图像的码流;获取所述目标视频中音频信息;将所述目标视频包括的多帧图像中每一帧图像对应所述基准帧图像的码流和所述音频信息通过所述第一无线信道发送。
- 根据权利要求5所述方法,其特征在于,所述目标图像为目标视频包括的多帧目标图像中的任意一个;所述方法还包括:获取所述目标视频包括的多帧目标图像中每一帧目标图像对应所述非基准帧图像的码流;将所述目标视频包括的多帧图像中每一帧图像对应所述非基准帧图像的码流通过所述第二无线信道发送。
- 一种低延迟信源信道联合编码方法,其特征在于,包括:对目标图像进行下采样,得到基准帧图像与非基准帧图像;根据当前无线信道的信道环境,确定所述基准帧图像和所述非基准帧图像对应的编码参数;根据所述编码参数,对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流;分别发送所述第一码流以及所述第二码流。
- 根据权利要求8所述方法,其特征在于,所述编码参数用于控制对应的图像在进行图像编码时按照目标码率生成对应的码流;所述根据所述编码参数,对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流,包括:将所述基准帧图像包括的每个所述基准帧图像按照所述目标码率进行帧内压缩,获得所述第一码流;将所述非基准帧图像与所述基准帧图像的残差按照所述目标码率进行所述帧内压缩,获得对应的所述第二码流。
- 根据权利要求8或9所述方法,其特征在于,所述信道环境包括信道的带宽、信号与干扰加噪声比、信噪比、接收的信号强度指示、占空比、比特率中的一个或多个;所述分别发送所述第一码流以及所述第二码流之前,还包括:根据所述信道环境,确定所述第一码流和/或所述第二码流分别对应的发送参数,其中,所述发送参数用于按照目标调制与编码策略信息和/或目标发射功率发送码流,所述第一码流的发送参数优于所述第二码流的发送参数。
- 根据权利要求8-10所述任意一项方法,其特征在于,所述无线信道包括第一无线信道和第二无线信道;所述分别发送所述第一码流以及所述第二码流,包括:将所述第一码流通过第一无线信道发送,将所述第二码流通过第二无线信道发送,其中,所述第一无线信道的服务质量保障机制高于所述第二无线信道。
- 根据权利要求11所述方法,其特征在于,所述目标图像为目标视频包括的多帧目标图像中的任意一个;所述方法还包括:获取所述目标视频包括的多帧目标图像中每一帧目标图像对应所述基准帧图像的码流;获取所述目标视频中音频信息;将所述目标视频包括的多帧图像中每一帧图像对应所述基准帧图像的码流和所述音频信息通过所述第一无线信道发送。
- 根据权利要求11所述方法,其特征在于,所述目标图像为目标视频包括的多帧目标图像中的任意一个;所述方法还包括:获取所述目标视频包括的多帧目标图像中每一帧目标图像对应所述非基准帧图像的码流;将所述目标视频包括的多帧图像中每一帧图像对应所述非基准帧图像的码流通过所述第二无线信道发送。
- 一种低延迟信源信道联合解码方法,其特征在于,包括:接收编码端发送的第一码流,所述第一码流为基准帧图像进行图像编码后获得的码流,其中,所述基准帧图像包括一个或多个对目标图像进行下采样后获得的图像;对所述第一码流进行图像解码,获得所述第一码流对应的所述基准帧图像;根据所述基准帧图像,重构所述目标图像。
- 根据权利要求14所述方法,其特征在于,所述目标图像为目标视频包括的多帧目标图像中的任意一个;所述根据所述基准帧图像,重构所述目标图像,包括:根据所述基准帧图像与所述目标图像相邻的一帧目标图像对应的基准帧图像和非基准帧图像中的至少一个,通过插值算法重构所述目标图像。
- 根据权利要求14所述方法,其特征在于,所述方法还包括:在接收所述第一码流后的预设时间段内,接收所述编码端发送的第二码流,其中,所述第二码流为非基准帧图像进行所述图像编码后获得的码流,所述非基准帧图像包括对所 述目标图像进行下采样后获得的除所述基准帧图像外剩余的图像;若所述第二码流残缺,将所述第二码流进行图像解码后,获得对应的残缺非基准帧图像;根据所述残缺非基准帧图像,确定所述残缺非基准帧图像的周边像素;所述根据所述基准帧图像,重构所述目标图像,包括:根据所述基准帧图像与所述周边像素,通过插值算法重构所述目标图像。
- 根据权利要求16所述方法,其特征在于,所述方法还包括:若所述第二码流完整,将所述第二码流进行图像解码后,获得对应的所述非基准帧图像;所述根据所述基准帧图像,重构所述目标图像,包括:将所述基准帧图像和所述非基准帧图像拼接,以重构所述目标图像。
- 一种低延迟信源信道联合编码装置,其特征在于,包括:编码器和发射机,其中,所述编码器,用于:对目标图像进行下采样,得到基准帧图像与非基准帧图像;对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流;所述发射机,用于:基于当前无线信道的信道环境,确定第一信道资源以及第二信道资源,分别利用所述第一信道资源发送所述第一码流以及利用所述第二信道资源发送所述第二码流,其中,所述第一信道资源优于所述第二信道资源。
- 一种低延迟信源信道联合编码装置,其特征在于,包括:编码器和发射机,其中,所述编码器,用于:对目标图像进行下采样,得到基准帧图像与非基准帧图像;根据当前无线信道的信道环境,确定所述基准帧图像和所述非基准帧图像对应的编码参数;根据所述编码参数,对所述基准帧图像和所述非基准帧图像分别进行图像编码,获得所述基准帧图像编码后的第一码流,和所述非基准帧图像编码后的第二码流;所述发射机,用于:分别发送所述第一码流以及所述第二码流。
- 一种计算机存储介质,其特征在于,所述计算机存储介质存储有计算机程序,该计算机程序被处理器执行时实现上述权利要求1-7或8-13任意一项所述的方法。
- 一种计算机程序,其特征在于,所述计算机程序包括指令,当所述计算机程序被计算机执行时,使得所述计算机执行如权利要求1-7或8-13中任意一项所述的方法。
- 一种低延迟信源信道联合编码***,其特征在于,包括编码端和解码端,其中,所述编码端用于执行如权利要求1-7或8-13所述的方法,所述解码端用于执行如权利要求14-17所述的方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2019/109220 WO2021056575A1 (zh) | 2019-09-29 | 2019-09-29 | 一种低延迟信源信道联合编码方法及相关设备 |
CN201980100614.8A CN114424552A (zh) | 2019-09-29 | 2019-09-29 | 一种低延迟信源信道联合编码方法及相关设备 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2019/109220 WO2021056575A1 (zh) | 2019-09-29 | 2019-09-29 | 一种低延迟信源信道联合编码方法及相关设备 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021056575A1 true WO2021056575A1 (zh) | 2021-04-01 |
Family
ID=75165457
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/109220 WO2021056575A1 (zh) | 2019-09-29 | 2019-09-29 | 一种低延迟信源信道联合编码方法及相关设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN114424552A (zh) |
WO (1) | WO2021056575A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024114475A1 (zh) * | 2022-11-30 | 2024-06-06 | 摩尔线程智能科技(北京)有限责任公司 | 一种视频转码方法及装置、电子设备、计算机可读存储介质和计算机程序产品 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116456094B (zh) * | 2023-06-15 | 2023-09-05 | 中南大学 | 一种分布式视频混合数字模拟传输方法及相关设备 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040179598A1 (en) * | 2003-02-21 | 2004-09-16 | Jian Zhou | Multi-path transmission of fine-granular scalability video streams |
CN102281436A (zh) * | 2011-03-15 | 2011-12-14 | 福建星网锐捷网络有限公司 | 无线视频传输方法、装置及网络设备 |
US20120092991A1 (en) * | 2010-10-15 | 2012-04-19 | Apple Inc. | Adapting transmission to improve qos in a mobile wireless device |
CN103580773A (zh) * | 2012-07-18 | 2014-02-12 | 中兴通讯股份有限公司 | 数据帧的传输方法及装置 |
CN107809662A (zh) * | 2017-11-06 | 2018-03-16 | 陕西师范大学 | 一种基于异构无线自组织网络的可分级视频传输方法及装置 |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1803302A4 (en) * | 2004-10-18 | 2007-11-07 | Samsung Electronics Co Ltd | DEVICE AND METHOD FOR ADJUSTING THE BIT RATE OF A BINARY TRAIN EVOLUTIVE CODE BY MULTILAYER |
KR100664929B1 (ko) * | 2004-10-21 | 2007-01-04 | 삼성전자주식회사 | 다 계층 기반의 비디오 코더에서 모션 벡터를 효율적으로압축하는 방법 및 장치 |
CN100588249C (zh) * | 2006-07-27 | 2010-02-03 | 腾讯科技(深圳)有限公司 | 调节视频质量的方法、***及终端 |
CN100584017C (zh) * | 2006-12-31 | 2010-01-20 | 联想(北京)有限公司 | 基于p2p网络的视频通信方法 |
CN101404759B (zh) * | 2008-10-30 | 2010-09-08 | 中山大学 | 一种用于数字视频监控***的网络自适应*** |
CN101448157A (zh) * | 2008-12-30 | 2009-06-03 | 杭州华三通信技术有限公司 | 一种视频编码方法和视频编码器 |
GB2499865B (en) * | 2012-03-02 | 2016-07-06 | Canon Kk | Method and devices for encoding a sequence of images into a scalable video bit-stream, and decoding a corresponding scalable video bit-stream |
WO2013139212A1 (en) * | 2012-03-21 | 2013-09-26 | Mediatek Singapore Pte. Ltd. | Method and apparatus for intra mode derivation and coding in scalable video coding |
CN102769747A (zh) * | 2012-06-29 | 2012-11-07 | 中山大学 | 一种基于并行迭代的分级分布式视频编解码方法及*** |
CN103716630B (zh) * | 2012-09-29 | 2017-02-22 | 华为技术有限公司 | 上采样滤波器的生成方法和装置 |
DE102014006080A1 (de) * | 2014-04-25 | 2015-10-29 | Unify Gmbh & Co. Kg | Verfahren und Vorrichtung zur Übermittlung von kodierten Mediendaten |
CN108496369A (zh) * | 2017-03-30 | 2018-09-04 | 深圳市大疆创新科技有限公司 | 视频传输、接收方法、***、设备及无人飞行器 |
CN109618188B (zh) * | 2018-12-19 | 2021-03-30 | 北京东土科技股份有限公司 | 视频数据的编码、转发方法、装置、设备及存储介质 |
-
2019
- 2019-09-29 CN CN201980100614.8A patent/CN114424552A/zh active Pending
- 2019-09-29 WO PCT/CN2019/109220 patent/WO2021056575A1/zh active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040179598A1 (en) * | 2003-02-21 | 2004-09-16 | Jian Zhou | Multi-path transmission of fine-granular scalability video streams |
US20120092991A1 (en) * | 2010-10-15 | 2012-04-19 | Apple Inc. | Adapting transmission to improve qos in a mobile wireless device |
CN102281436A (zh) * | 2011-03-15 | 2011-12-14 | 福建星网锐捷网络有限公司 | 无线视频传输方法、装置及网络设备 |
CN103580773A (zh) * | 2012-07-18 | 2014-02-12 | 中兴通讯股份有限公司 | 数据帧的传输方法及装置 |
CN107809662A (zh) * | 2017-11-06 | 2018-03-16 | 陕西师范大学 | 一种基于异构无线自组织网络的可分级视频传输方法及装置 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024114475A1 (zh) * | 2022-11-30 | 2024-06-06 | 摩尔线程智能科技(北京)有限责任公司 | 一种视频转码方法及装置、电子设备、计算机可读存储介质和计算机程序产品 |
Also Published As
Publication number | Publication date |
---|---|
CN114424552A (zh) | 2022-04-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
TWI663880B (zh) | 色度預測的方法和設備 | |
JP5882547B2 (ja) | シーンの変化に伴うピクチャ内の符号化及び送信パラメータの好適化 | |
US9071841B2 (en) | Video transcoding with dynamically modifiable spatial resolution | |
US7924917B2 (en) | Method for encoding and decoding video signals | |
TWI647946B (zh) | 一種圖像編解碼方法及裝置 | |
TWI505694B (zh) | 編碼器及編碼方法 | |
US9414086B2 (en) | Partial frame utilization in video codecs | |
US11743475B2 (en) | Advanced video coding method, system, apparatus, and storage medium | |
TW201931853A (zh) | 具有聯合像素/變換為基礎之量化之視頻寫碼之量化參數控制 | |
CN102396225B (zh) | 用于可靠实时传输的图像和视频的双模式压缩 | |
US10542265B2 (en) | Self-adaptive prediction method for multi-layer codec | |
CN103493481A (zh) | 基于场景的适应性比特率控制 | |
US20220295071A1 (en) | Video encoding method, video decoding method, and corresponding apparatus | |
WO2021185257A1 (zh) | 图像编码方法、图像解码方法及相关装置 | |
US20080205513A1 (en) | Method and system for upsampling a spatial layered coded video image | |
CA3057894C (en) | Video compression using down-sampling patterns in two phases | |
WO2021056575A1 (zh) | 一种低延迟信源信道联合编码方法及相关设备 | |
US20230017934A1 (en) | Image encoding and decoding methods and apparatuses | |
US11582462B1 (en) | Constraint-modified selection of video encoding configurations | |
CN115866297A (zh) | 视频处理方法、装置、设备及存储介质 | |
KR20060043050A (ko) | 영상 신호의 인코딩 및 디코딩 방법 | |
KR20220063063A (ko) | 인공지능 부호화 및 인공지능 복호화를 수행하기 위한 방법 및 장치 | |
US20150078433A1 (en) | Reducing bandwidth and/or storage of video bitstreams | |
Jung | Comparison of video quality assessment methods | |
US20230196505A1 (en) | Artificial intelligence-based image providing apparatus and method, and artificial intelligence-based display apparatus and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19946291 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19946291 Country of ref document: EP Kind code of ref document: A1 |