CA3187271A1 - Frame-rate scalable video coding - Google Patents

Frame-rate scalable video coding Download PDF

Info

Publication number
CA3187271A1
CA3187271A1 CA3187271A CA3187271A CA3187271A1 CA 3187271 A1 CA3187271 A1 CA 3187271A1 CA 3187271 A CA3187271 A CA 3187271A CA 3187271 A CA3187271 A CA 3187271A CA 3187271 A1 CA3187271 A1 CA 3187271A1
Authority
CA
Canada
Prior art keywords
shutter
interval
frame
sub
flag
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CA3187271A
Other languages
French (fr)
Other versions
CA3187271C (en
Inventor
Robin Atkins
Peng Yin
Taoran Lu
Fangjun PU
Sean Thomas MCCARTHY
Walter J. Husak
Tao Chen
Guan-Ming Su
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dolby Laboratories Licensing Corp
Original Assignee
Dolby Laboratories Licensing Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dolby Laboratories Licensing Corp filed Critical Dolby Laboratories Licensing Corp
Priority to CA3192296A priority Critical patent/CA3192296A1/en
Priority to CA3231109A priority patent/CA3231109A1/en
Priority to CA3231105A priority patent/CA3231105A1/en
Publication of CA3187271A1 publication Critical patent/CA3187271A1/en
Application granted granted Critical
Publication of CA3187271C publication Critical patent/CA3187271C/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/31Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the temporal domain
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/187Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46Embedding additional information in the video signal during the compression process
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Holo Graphy (AREA)
  • Television Systems (AREA)
  • Particle Accelerators (AREA)
  • Mechanical Light Control Or Optical Switches (AREA)
  • Indication In Cameras, And Counting Of Exposures (AREA)
  • Liquid Crystal (AREA)
  • Studio Devices (AREA)
  • Exposure Control For Cameras (AREA)

Abstract

Methods and systems for frame rate scalability are described. Support is provided for input and output video sequences with variable frame rate and variable shutter angle across scenes, or for input video sequences with fixed input frame rate and input shutter angle, but allowing a decoder to generate a video output at a different output frame rate and shutter angle than the corresponding input values. Techniques allowing a decoder to decode more computationally-efficiently a specific backward compatible target frame rate and shutter angle among those allowed are also presented.

Description

FRAME-RATE SCALABLE VIDEO CODING
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application claims the benefit of priority from U.S.
Patent Application 16/901,911, filed on June 15, 2020, and U.S. Patent Application No.
17/212,701, filed on March 25, 2021, each of which is incorporated by reference in its entirety.
TECHNOLOGY
[0002] The present document relates generally to images. More particularly, an embodiment .. of the present invention relates to frame-rate scalable video coding.
BACKGROUND
[0003] As used herein, the term 'dynamic range' (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (blacks) to brightest whites (highlights). In this sense, DR relates to a 'scene-referred' intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a 'display-referred' intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably.
[0004] As used herein, the term high dynamic range (HDR) relates to a DR
breadth that spans the 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR
over which a human may simultaneously perceive an extensive breadth in intensity range may be somewhat truncated, in relation to HDR.
[0005] In practice, images comprise one or more color components (e.g., luma Y and chroma Cb and Cr) wherein each color component is represented by a precision of n-bits per pixel (e.g., n=8). Using linear luminance coding, images where n < 8 (e.g., color 24-bit JPEG
images) are considered images of standard dynamic range (SDR), while images where n> 8 may be considered images of enhanced dynamic range. HDR images may also be stored and .. distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.
[0006] Currently, distribution of video high dynamic range content, such as Dolby Vision from Dolby laboratories or HDR10 in Blue-Ray, is limited to 4K resolution (e.g., 4096 x 2160 or 3840 x 2160, and the like) and 60 frames per second (fps) by the capabilities of many playback devices. In future versions, it is anticipated that content of up to 8K
resolution (e.g., 7680 x 4320) and 120 fps may be available for distribution and playback. It is desirable that future content types will be compatible with existing playback devices in order to simplify an HDR
playback content ecosystem, such as Dolby Vision. Ideally, content producers should be able to adopt and distribute future HDR technologies without having to also derive and distribute special versions of the content that are compatible with existing HDR devices (such as HDR10 or Dolby Vision). As appreciated by the inventors here, improved techniques for the scalable distribution of video content, especially HDR content, are desired.
[0007] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued.
Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
[0009] FIG. 1 depicts an example process for a video delivery pipeline;
[00010] FIG. 2 depicts an example process of combining consecutive original frames to render a target frame rate at a target shutter angle according to an embodiment of this invention;
[00011] FIG. 3 depicts an example representation of an input sequence with variable input frame rate and variable shuttle angle in a container with a fixed frame rate according to an embodiment of this invention; and
[00012] FIG. 4 depicts an example representation for temporal scalability at various frame rates and shutter angles with backwards compatibility according to an embodiment of this invention.

DESCRIPTION OF EXAMPLE EMBODIMENTS
[00013] Example embodiments that relate to frame-rate scalability for video coding are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of present invention. It will be apparent, however, that the various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating embodiments of the present invention.
SUMMARY
[00014] Example embodiments described herein relate to frame rate scalability in video coding. In an embodiment, a system with a processor receives a coded bitstream comprising coded video frames, wherein one or more coded frames are encoded in a first frame rate and a first shutter angle. The processor receives a first flag indicating the presence of a group of coded frames to be decoded at a second frame rate and a second shutter angle, it accesses from the coded bitstream values of the second frame rate and the second shutter angle for the group of coded frames, and generates decoded frames at the second frame rate and the second shutter angle based on the group of coded frames, the first frame rate, the first shutter angle, the second frame rate and the second shutter angle.
[00015] In a second embodiment, a decoder with a processor:
receives a coded bitstream comprising groups of coded video frames, wherein all coded video frames in the coded bitstream are encoded in a first frame rate;
receives a number of combined frames N;
receives a value for a baseline frame rate;
accesses a group of N consecutive coded frames, wherein the i-th coded frame in the group of N consecutive coded frames, wherein i= 1, 2, ...N, represents an average of up to i input video frames encoded in an encoder at the baseline frame rate and an i-th shutter angle based on a first shutter angle and the first frame rate;

accesses from the coded bitstream or from user input values for a second frame rate and a second shutter angle, for decoding the group of N consecutive coded frames in the second frame rate and the second shutter angle; and generates decoded frames at the second frame rate and the second shutter angle based on .. the group of N consecutive coded frames, the first frame rate, the first shutter angle, the second frame rate, and the second shutter angle.
[00016] In a third embodiment, an encoded video stream structure comprises:
an encoded picture section including an encoding of a sequence of video pictures; and a signaling section including an encoding of:
a shutter interval time-scale parameter indicating the number of time units passing in one second;
a shutter interval clock-ticks parameter indicating a number of time units of a clock operating at the frequency of the shutter interval time-scale parameter, wherein the shutter interval clock-ticks parameter divided by the shutter interval time-scale parameter indicates an exposure duration value;
a shutter-interval-duration flag indicating whether exposure duration information is fixed for all temporal sub-layers in the encoded picture section; and if the shutter-interval-duration flag indicates that the exposure duration information is fixed, then a decoded version of the sequence of video pictures for all the temporal sub-layers in the encoded picture section is decoded by computing the exposure duration value based on the shutter interval time-scale parameter and the shutter interval clock-ticks parameter, else the signaling section includes one or more arrays of sub-layer parameters, wherein values in the one or more arrays of sub-layer parameters combined with the shutter interval time-scale parameter are used to compute for each sub-layer a corresponding sub-layer exposure duration value for displaying a decoded version of the temporal sub-layer of the sequence of video pictures.

EXAMPLE VIDEO DELIVERY PROCESSING PIPELINE
[00017] FIG. 1 depicts an example process of a conventional video delivery pipeline (100) showing various stages from video capture to video content display. A sequence of video frames (102) is captured or generated using image generation block (105). Video frames (102) may be digitally captured (e.g. by a digital camera) or generated by a computer (e.g.
using computer animation) to provide video data (107). Alternatively, video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide video data (107). In a production phase (110), video data (107) is edited to provide a video production stream (112).
[00018] The video data of production stream (112) is then provided to a processor at block (115) for post-production editing. Block (115) post-production editing may include adjusting or modifying colors or brightness in particular areas of an image to enhance the image quality or achieve a particular appearance for the image in accordance with the video creator's creative intent. This is sometimes called "color timing" or "color grading." Other editing (e.g. scene selection and sequencing, image cropping, addition of computer-generated visual special effects, judder or blur control, frame rate control, etc.) may be performed at block (115) to yield a final version (117) of the production for distribution. During post-production editing (115), video images are viewed on a reference display (125). Following post-production (115), video data of final production (117) may be delivered to encoding block (120) for delivering downstream to decoding and playback devices such as television sets, set-top boxes, movie theaters, and the .. like. In some embodiments, coding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate coded bit stream (122). In a receiver, the coded bit stream (122) is decoded by decoding unit (130) to generate a decoded signal (132) representing an identical or close approximation of signal (117).
The receiver may be attached to a target display (140) which may have completely different characteristics than the reference display (125). In that case, a display management block (135) may be used to map the dynamic range of decoded signal (132) to the characteristics of the target display (140) by generating display-mapped signal (137).

SCALABLE CODING
[00019] Scalable coding is already part of a number of video coding standards, such as, MPEG-2, AVC, and HEVC. In embodiments of this invention, scalable coding is extended to improve performance and flexibility, especially as it relates to very high resolution HDR content.
[00020] As used herein, the term "shutter angle" denotes an adjustable shutter setting which controls the proportion of time that film is exposed to light during each frame interval. For example, in an embodiment shutter angle exposure time = __________________________________________________________________________ (1) 360 frame interval. [00021] The term comes from legacy, mechanical, rotary shutters; however, modern digital cameras can also adjust their shutter electronically. Cinematographers may use the shutter angle to control the amount of motion blur or judder that is recorded in each frame.
Note that instead of using "exposure time" one may also use alternative terms, like "exposure duration, ""shutter interval," and "shutter speed." Similarly, instead of using "frame interval"
one may use the term "frame duration." Alternatively, one may replace "frame interval" with "1/frame rate." The value of exposure time is typically less than or equal to the duration of a frame. For example, a shutter angle of 180 degrees indicates that the exposure time is half of the frame duration. In some situations, exposure time may be greater than the frame duration of coded video, for example, when the encoded frame rate is 120 fps and the frame rate of the associated video content prior to encoding and display is 60 fps.
[00022] Consider, without limitation, an embodiment where original content is shot (or generated) at an original frame rate (e.g., 120 fps) with a shutter angle of 360 degrees. Then, in a receiving device, one can render video output at a variety of frame rates equal to or lower than the original frame rate by judicial combination of the original frames, e.g., by averaging or other known in the art operations.
[00023] The combining process may be performed with non-linear encoded signals, (e.g., using gamma, PQ or HLG), but best image quality is obtained by combining frames in the linear light domain by first, converting the non-linear encoded signals into linear-light representations, next, combining the converted frames, and finally re-encoding the output with the non-linear transfer function. This process provides a more accurate simulation of a physical camera exposure than combining in the non-linear domain.

[00024] In general terms, the process of combining frames can be express in terms of the original frame rate, the target frame rate, the target shutter angle, and the number of frames to be combined as:
n_fra mes = (target_shutter_angle/360)*(original_fra me_rate/target_fra me_rate), (2) which is equivalent to target_shutter_angle = 360*n_frames*(target_frame_rate/original_frame_rate), (3) where n_frames is the number of combined frames, original_frame_rate is the frame rate of the original content, target_frame_rate is the frame rate to be rendered (where, target_frame_rate < original_frame_rate), and target_shutter_angle indicates the amount of desired motion blur.
In this example, the maximum value of target_shutter_angle is 360 degrees and corresponds to the maximal motion blur. The minimum value of target_shutter_angle can be expressed as 360 *(target_frame_rate/original_frame_rate) and corresponds to minimal motion blur. The maximum value of n_frames can be expressed as (original_frame_rate/target_frame_rate). The values of target_frame_rate and target_shutter_angle should be selected such that the value of n_frames is a non-zero integer.
[00025] In the special case that the original frame rate is 120 fps, equation (2) can be rewritten as n_frames = target_shutter_angle/(3*target_frame_rate), (4) which is equivalent to target_shutter_angle = 3*n_frames*target_frame_rate.
(5) The relationships between the values of target_frame_rate, n_frames, and target_shutter_angle are shown in Table 1 for the case of original_frame_rate = 120 fps. In Table 1, "NA" indicates that the corresponding combination of a target frame rate and the number of combined frames is not allowed.
Table 1:Relationship among target frame rate, number of frames combined, and target shutter angle, for an original frame rate of 120 fps.
Target Number of Frames Combined Frame Rate 5 4 3 2 1 (fps) Target Shutter Angle (degrees) [00026] FIG. 2 depicts an example process of combining consecutive original frames to render a target frame rate at a target shutter angle according to an embodiment. Given an input sequence (205) at 120 fps and a shutter angle of 360 degrees, the process generates an output video sequence (210) at 24 fps and a shutter angle of 216 degrees by combining three of the input frames in a set of five consecutive frames (e.g., the first three consecutive frames), and dropping the other two. Note that in some embodiments, output frame-01 of (210) may be generated by combining alternative input frames (205), such as frames 1, 3, and 5, or frames 2,4, and 5, and the like; however, it is expected that combining consecutive frames will yield video output of better quality.
[00027] It is desirable to support original content with variable frame rate, for example, to manage artistic and stylistic effect. It is also desirable that the variable input frame rate of the original content is packaged in a "container" that has a fixed frame rate to simplify content production, exchange, and distribution. As an example, three embodiments on how to represent the variable frame rate video data in a fixed frame rate container are presented. For purposes of clarity and without limitation, the following descriptions use fixed 120 fps container, but the approaches can easily be extended to an alternative frame rate container.

First Embodiment (Variable Frame Rate) [00028] The first embodiment is an explicit description of original content having variable (non-constant) frame rate packaged in a container having constant frame rate.
For example, original content that has different frames rate, say, at 24, 30, 40, 60, or 120 fps, for different scenes, may be packaged in a container having a constant frame rate of 120 fps. For this example, each input frame can be duplicated either 5x, 4x, 3x, 2x, or lx times to package it into a common 120 fps container.
[00029] FIG. 3 depicts an example of an input video sequence A with variable frame rate and variable shutter angle which is represented in a coded bitstream B with a fixed frame rate. Then, in a decoder, the decoder reconstructs output video sequence C at the desired frame rate and shutter angle, which may change from scene to scene. For example, as depicted in FIG. 3, to construct sequence B, some of the input frames are duplicated, some are coded as is (with no duplication), and some are copied four times. Then, to construct sequence C, any one frame from each set of duplicate frames is selected to generate output frames, matching the original frame rate and shutter angle.
[00030] In this embodiment, metadata is inserted in the bitstream to indicate the original (base) frame rate and shutter angle. The metadata may be signaled using high level syntax such as a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), a Slice or Tile Group header, and the like. The presence of metadata enables encoders and decoders to perform beneficial functions, such as:
a) An encoder can ignore duplicated frames, thereby increasing encoding speed and simplifying processing. For example, all coding tree units (CTUs) in duplicated frames can be encoded using SKIP mode and reference index 0 in LIST 0 of the reference frames, which refers to a decoded frame from which duplicated frames are copied.
b) A decoder can bypass decoding of duplicate frames thereby simplifying processing. For example, metadata in the bitstream can indicate that a frame is a duplicate of a previously decoded frame that the decoder can reproduce by copying and without decoding the new frame.
c) A playback device can optimize downstream processing by indicating the base frame rate, for example by adjusting frame rate conversion or noise reduction algorithms.

[00031] This embodiment enables an end user to view rendered content at the frame rates intended by the content creators. This embodiment does not provide for backwards compatibility with devices that do not support the frame rate of the container, e.g., 120 fps.
[00032] Tables 2 and 3 depict example syntax of raw byte sequence payload (RBSB) for a sequence parameter set and Tile Group header, where the proposed new syntax elements are depicted in an italic font. The remaining syntax follows the syntax in the proposed specification of the Versatile Video Codec (VVC) (Ref.[2]).
[00033] As an example, in SPS (see Table 2), one may add a flag to enable variable frame rate.
sps_vfr_enabled_flag equal to 1 specifies that the coded video sequence (CVS) may contain variable frame rate content. sps vfr enabled flag equal to 0 specifies that the CVS contains fixed frame rate content.
In the tile group header() (see Table 3), tile group vrf info present flag equal to 1 specifies the syntax elements tile group true fr and tile group shutterangle are present in the syntax. tile group vrf info present flag equal to 0 specifies the syntax elements tile group true fr and tile group shutterangle are not present in the syntax. When tile group vrf info present flag is not present, it is inferred to be 0.
tile group true fr indicates the true frame rate of the video data carried in this bitstream.
tile group shutterangle indicates the shutter angle corresponding to the true frame rate of the video data carried in this bitstream.
tile group skip flag equal to 1 specifies that the current tile group is copied from another tile group. tile group skip flag equal to 0 specifies that the current tile group is not copied from another tile group.
tile group copy pic order cnt lsb specifies the picture order count modulo MaxPicOrderCntLsb for the previously decoded picture which the current picture copies from when tile group skip flag is set to 1.

Table 2: Example parameter set RBSP syntax for content with variable frame-rate seq_parameter_set_rbsp( ) {
Descriptor sps_max_sub_layers_minusl u(3) sps_reserved_zero_5bits u(5) profile_tier_leveh sps_max_sub_layers_minusl ) sps_seq_parameter_set_id ue(v) ...
sps_vfr_enabled_flag u (1) ...
sps_extension_flag u(1) if( sps_extension_flag ) while( more_rbsp_data( ) ) sps_extension_data_flag u(1) rbsp_trailing_bits( ) Table 3: Example of Tile Group header syntax with support for content with variable frame rate tile_group_header( ) {
Descriptor tile_group_pic_parameter_set_id ue(v) if( NumTilesInPic > 1) {
tile_group_address u(v) num_tiles_in_tile_group_minusl ue(v) tile_group_type ue(v) tile_group_pic_order_cnt_lsb u(v) if( sps_vfr_enabled_flag ) {
tde_group_vfr_info_present_flag u(1) if (tile_group_vfr_info_present_flag) {

tile_group_true_fr u(9) tile_group_shutterangle u(9) ]
tile_group_skip_flag u(1) I
if( tile_group_skip_flag ) tile_group_copy_pic_order_cnt_lsb u(v) else[
ALL OTHER TILE_GROUP_SYNTAX

if( num_tiles_in_tile_group_minusl > 0) I
offset_len_minusl ue(v) for( i = 0; i < num_tiles_in_tile_group_minusl; i++) entry_point_offset_minusl Iii]
u(v) byte_alignment( ) Second Embodiment ¨ Fixed frame rate container [00034] The second embodiment enables the use case in which original content having a fixed frame rate and shutter angle may be rendered by a decoder at an alternative frame rate and variable simulated shutter angle, such as illustrated in FIG. 2. For example, in the case that original content has a frame rate of 120 fps and a shutter angle of 360 degrees (meaning the shutter is open 1/120 second), a decoder can render out multiple frame rates that are less than or equal to 120 fps. For example, as described in Table 1, to decode 24 fps with a 216-degrees simulated shutter angle, the decoder may combine three decoded frames and display at 24 fps.
Table 4 expands upon Table 1 and illustrates how to combine different numbers of encoded frames to render at the output target frame rates and the desired target shutter angles. Combining the frames may be performed by simple pixel averaging, by weighted pixel averaging, where pixels from a certain frame may be weighted more than pixels of other frames and the sum of all weights sums to one, or by other filter interpolation schemes known in the art. In Table 4, the function Ce(a,b) denotes the combination of encoded frames a to b, where the combining can be performed by averaging, weighted averaging, filtering, and the like.
Table 4:Example of combining input frames at 120 fps to generate output frames at target fps and shutter angle values Input sl s2 s3 s4 s5 s6 s7 s8 s9 slO
Enc. el e2 e3 e4 e5 e6 e7 e8 e9 e10 Dec.
120fps @360 el e2 e3 e4 e5 e6 e7 e8 e9 e10 Dec.
60fps @360 Ce(1,2) Ce(3,4) Ce(5,6) Ce(7,8) Ce(9,10) @180 el e3 e5 e7 e9 Dec.
40fps @360 Ce(1,3) Ce(4,6) Ce(7,9) Ce(10,12) @240 Ce(1,2) Ce(4,5) Ce(7,8) Ce(10,11) @120 el e4 e7 el0 Dec.
30fps @360 Ce(1,4) Ce(5,8) Ce(9,12) @270 C(1,3) Ce(5,7) Ce(9,11) @180 Ce(1,2) Ce(5,6) Ce(9,10) @90 el e5 e9 Dec.
24fps @360 Ce(1,5) Ce(6,10) @288 Ce(1,4) Ce(6,9) @216 Ce(1,3) Ce(6,8) @144 Ce(1,2) Ce(6,7) @72 el e6 [00035] When the value of the target shutter angle is less than 360 degrees, the decoder can combine different sets of decoded frames. For example, from Table 1, given an original stream of 120 fps @ 360-degrees, to generate a stream at 40 fps and a 240-degrees shutter angle, a decoder needs to combine two frames out of three possible frames. Thus, it may combine either the first and the second frames or the second and the third frames. The choice of which frames to combine may be described in terms of a "decoding phase" expressed as:
decode_phase = decode_phase_idx*(360/n_frames), (6) where decode_phase_idx indicates the offset index within a set of sequential frames having index values in [0, n_frames_max-1], where n_frames is given by equation (2), and n_fra mes_max = ong_frame_rate/target_frame_rate.
(7) [00036] In general, decode_phase_idx ranges from [0, n_frames_max-n_frames].
For example, for an original sequence at 120 fps and a 360 degrees shutter angle, for the target frame rate of 40 fps at a 240 degrees shutter angle, n_frames_max = 120/40 = 3. From equation (2), n_frames = 2, thus decode_phase_idx ranges from [0, 1]. Thus, decode_phase_idx = 0 indicates selecting frames with index 0 and 1, and decode_phase_idx = 1 indicates selecting frames with index 1 and 2.
[00037] In this embodiment, the rendered variable frame rate intended by the content creator may be signaled as metadata, such as a supplemental enhancement information (SET) message or as video usability information (VUI). Optionally, the rendered frame rate may be controlled by the receiver or a user. An example of frame rate conversion SET messaging that specifies the preferred frame rate and shutter angle of the content creator is shown in Table 5. The SET
message can also indicate if combining frames is performed in the coded signal domain (e.g., gamma, PQ, etc.) or the linear light domain. Note that postprocessing requires a frame buffer in addition to the decoder picture buffer (DPB). The SET message may indicate how many extra frame buffers are needed, or some alternative method for combining frames. For example, to reduce complexity, frames may be recombined at reduced spatial resolution.
[00038] As depicted in Table 4, at certain combinations of frame rates and shutter angles (e.g., at 30 fps and 360 degrees or at 24 fps and 288 or 360 degrees) a decoder may need to combine more than three decoded frames, which increases the number of buffer space required by the decoder. To reduce the burden of extra buffer space in the decoder, in some embodiments, certain combinations of frame rates and shutter angles may be off limits to the set of allowed .. decoding parameters (e.g., by setting appropriate coding Profiles and Levels).
[00039] Considering again, as an example, the case of playback at 24 fps, a decoder may decide to display the same frame five times to be displayed at 120 fps output frame rate. This is exactly the same as showing the frame a single time at 24 fps output frame rate. The advantage of keeping a constant output frame rate is that a display can run at a constant clock speed, which makes all the hardware much simpler. If the display can dynamically vary the clock speed then it may make more sense to only show the frame once (for 1/24th of a second), instead of repeating the same frame five times (each 1/120th of a second). The former approach may result in slightly higher picture quality, better optical efficiency, or better power efficiency.
Similar considerations are also applicable to other frame rates.
[00040] Table 5 depicts an example of a frame rate conversion SET messaging syntax according to an embodiment.

Table 5: Example of SET message syntax allowing frame-rate conversion framerate_conversion( payloadSize ) { Descriptor framerate_conversion_cancel_flag u(1) if( !frame_conversion_cancel_flag ) {
base_frame_rate u(9) base_shutter_angle u(9) decode_phase_idx_present_flag u(1) if ( decode_phase_idx_present_flag ) {
decode_phase_idx u(3) conversion_domain_idc u(1) num_frame_buffer u(3) framerate_conversion_persistence_flag u(1) framerate conversion cancel flag equal to 1 indicates that the SET message cancels the persistence of any previous frame rate conversion SET message in output order.
framerate conversion cancel flag equal to 0 indicates that framerate conversion information follows.
base frame rate specifies the desired frame rate.
base shutter angle specifies the desired shutter angle.
decode phase idx present flag equal to 1 specifies that decoding phase information is present.
decode phase idx present flag equal to 0 specifies that decoding phase information is not present.
decode_phase_idx indicates the offset index within a set of sequential frames having index values 0..(n frames max-1) where n frames max = 120/base frame rate. The value of decode_phase idx shall be in the range of 0..(n frames max-n frames), where n frames =
base shutter angle/(3*base frame rate). When decode_phase idx is not present, it is inferred to be O.

conversion domain idc equal to 0 specifies that frame combination is performed in linear domain. conversion domain idc equal to 1 specifies that frame combination is performed in non-linear domain.
num frame buffers specifies the additional number of frame buffers (not counting DPB).
framerate conversion persistence flag specifies the persistence of the frame rate conversion SET message for the current layer. framerate conversion persistence flag equal to 0 specifies that the framerate conversion SET message applies to the current decoded picture only. Let picA
be the current picture. framerate conversion persistence flag equal to 1 specifies that the frame rate conversion SET message persists for the current layer in output order until one or more of the following conditions are true:
¨ A new coded layer-wise video sequence (CLVS) of the current layer begins.
¨ The bitstream ends.
¨ A picture picB in the current layer in an access unit containing a framerate conversion SET
message that is applicable to the current layer is output for which PicOrderCnt( picB ) is greater than PicOrderCnt( picA ), where PicOrderCnt( picB ) and PicOrderCnt( picA ) are the PicOrderCntVal values of picB and picA, respectively, immediately after the invocation of the decoding process for picture order count for picB.
Third Embodiment ¨ Input encoded at multiple shutter angles .. [00041] A third embodiment is a coding scheme that allows the extraction of sub-frame rates from the bitstream, thus supporting backward compatibility. In HEVC, this is achieved by temporal scalability. Temporal-layer scalability is enabled by assigning different values to a temporal id syntax element for the decoded frames. The bitstream can thereby be extracted simply on the basis of temporal id values. However, the HEVC-style approach to temporal scalability does not enable rendering output frame rates with different shutter angles. For example, a 60 fps base frame rate extracted from an 120 fps original will always have a shutter angle of 180 degrees.
[00042] In ATSC 3.0, an alternative method is described in which frames at 60 fps having a 360 degrees shutter angles are emulated as a weighted average of two 120 fps frames. The emulated 60 fps frames are assigned temporal id value of 0 and are combined with alternating original 120 fps frames assigned temporal id value 1. When 60 fps is needed, the decoder only needs to decode frames with temporal id 0. When 120 fps is needed, the decoder may subtract each temporal id = 1 frame (i.e., a 120 fps frame) from a scaled version of each corresponding temporal id = 0 frame (i.e., emulated 60 fps frame) to recover the corresponding original 120 fps frame that was not transmitted explicitly, thereby reconstituting all the original 120 fps frames.
[00043] In embodiments of this invention, a new algorithm that supports multiple target frame rates and target shutter angles in a manner that is backward compatible (BC) is described. The proposal is to preprocess the original 120 fps content at a base frame rate at several shutter angles. Then, at the decoder, other frame rates at various other shutter angles can be simply derived. The ATSC 3.0 approach can be thought of as a special case of the proposed scheme, where frames with temporal id=0 carry frames at 60fps@360 shutter angle and frames with temporal id=1 carry frames at 60fps@180 shutter angle.
[00044] As a first example, as depicted in FIG. 4, consider an input sequence at 120 fps and a 360 shutter angle that is used to encode a sequence with a base layer frame rate of 40 fps and shutter angles at 120, 240, and 360 degrees. In this scheme the encoder computes new frames by combining up to three of the original input frames. For example, encoded frame 2 (En-2) representing the input at 40 fps and 240 degrees is generated by combining input frames 1 and 2, and encoded frame 3 (En-3) representing the input at 40 fps and 360 degrees is generated by combining frame En-2 to input frame 3. In the decoder, to reconstruct the input sequence, decoded frame 2 (Dec-2) is generated by subtracting frame En-1 from frame En-2, and decoded frame 3 (Dec-3) is generated by subtracting frame En-2 from frame En-3. The three decoded frames represent an output at base frame rate of 120 fps and shutter angle 360 degrees.
Additional frame rates and shutter angles can be extrapolated using the decoded frames as depicted in Table 6. In Table 6, the function Cs(a,b) denotes the combination of input frames a to b, where the combining can be performed by averaging, weighted averaging, filtering, and the like.

Table 6: Example of frame combination with a baseline of 40 fps Input Frames 120fps @360 sl s2 s3 s4 s5 s6 s7 s8 s9 Encoded Frames el= e2= e3 = e5= e6= e8 = e9 =
120fps sl Cs(1,2) Cs(1,3) e4=54 Cs(4,5) Cs(4,6) e7=57 Cs(7,8) Cs(7,9) Decode 120fps el= e2-el = e3-e2= e5-e4= e6-e4 = e8-e7= e9-e8 @360 sl s2 s3 e4 = s4 s5 s6 e7 = s7 s8 =s9 Decode 60fps e3-e2+e4 e9-@360 e2 =Cs(3,4) e6-e4=Cs(5,6) e8 = Cs(7,8) e8+e10 @180 el e3-e2=53 e5-e4=55 e7 e9-e8 Decode 40fps @360 e3 =Cs(1,3) e6 e9 @240 e2=Cs(1,2) e5 e8 @120 el=s1 e4 e7 Decode 30fps e9-@360 e3+e4 = Cs(1,4) e6-e4+e8 = Cs(5,8) e8+e12 e9-@270 e3 = Cs(1,3) e6-e5+e7 = Cs(5,7) e8+ell e9-@180 e2 = Cs(1,2) e6-e4=Cs(5,6) e8+e10 @90 el e5-e4 = s5 e9-e8 Decode 24fps @360 e3+e5 = Cs(1,5) e6-e5+e9+e10= Cs(6,10) @288 e3+e4 = Cs(1,4) e6-e5+e9 = Cs(6,9) @216 e3 = Cs(1,3) e6-e5+e8 = Cs(6,8) @144 e2 = Cs(1,2) e6-e5+e7=Cs(6,7) @72 el = Si e6-e5 = s6 [00045] An advantage of this approach is that, as depicted in Table 6, all the 40 fps versions can be decoded without any further processing. Another advantage is that other frame rates can be derived at various shutter angles. For example, consider a decoder decoding at 30 fps and a shutter angle of 360. From Table 4, the output corresponds to the sequence of frames generated by Ce(1,4) = Cs(1,4), Cs(5,8), Cs(9,12), and the like, which matches the decoding sequence depicted in Table 6 as well; however, in Table 6, Cs(5,8) = e6-e4+e8. In an embodiment, look-up tables (LUTs) can be used to define how the decoded frames need to be combined to generate an output sequence at the specified output frame rate and emulated shutter angle.
[00046] In another example, it is proposed to combine up to five frames in the encoder in order to simplify the extraction of the 24 fps base layer at shutter angles of 72, 144, 216, 288, and 360 degrees, as shown below. This is desirable for movie content that is best presented at 24fps on legacy televisions.
Table 7: Example of frame combination with a baseline of 24 fps Input Frames 120fps @360 sl s2 s3 s4 s5 s6 s7 s8 s9 Enc. el= e2= e6= e7= e8= e9=
frames sl Cs(1,2) e3=Cs(1,3) e4=Cs(1,4) e5=Cs(1,5) s6 Cs(6,7) Cs(6,8) Cs(6,9) Decode 120fps @360 el e2-el e3-e2 e4-e3 e5-e4 e6 e7-e6 e8-e7 e9-e8 Decode 60fps @360 e2 e4-e2 e5-e4+e6 e8-e6 e10-e8 @180 el e3-e2 e5-e4 e7-e6 e9-e8 Decode 40fps @360 e3 e5-e3+e6 e9-e6 @240 e2 e5-e3 e8-e6 @120 el e4-e3 e7-e6 Decode 30fps e10-@360 e4 e5-e4+e8 e8+e12 e10-@270 e3 e5-e4+e7 e8+ell @180 e2 e5-e4+e6 e10-e8 @90 el e5-e4 e9-e8 Decode 24fps @360 e5 el0 @288 e4 e9 @216 e3 e8
- 21 -@144 e2 e7 @72 el e6 [00047] As depicted in Table 7, if the decoding frame rate matches the baseline frame rate (24 fps), then, in each group of five frames (e.g., el to e5) a decoder can simply select the one frame at the desired shutter angle (e.g., e2 for a shutter angle at 144 degrees). To decode at a different frame rate and a specific shutter angle, the decoder will need to determine how to properly combine (say, by addition or subtraction) the decoded frames. For example, to decode at 30 fps and a shutter angle of 180 degrees, the following steps may be followed:
a) The decoder may consider a hypothetical encoder transmitting at 120 fps and 360 degrees without any consideration for backward compatibility, then, from Table 1, the decoder needs to combine 2 out of 4 frames to generate the output sequence at the desired frame rate and shutter angle. For example, as depicted in Table 4, the sequence includes, Ce(1,2) =
Avg(sl, s2), Ce(5,6) = Avg(55, s6), and the like, where Avg(sl,52) may denote averaging of frames sl and s2.
b) Given that by definition the encoded frames can be expressed as el = sl, e2 = Avg(sl, s2), e3 = Avg(sl, s3), and the like, one can easily derive that the sequence of frames in step a) can also be expressed as:
= Ce(1,2) = Avg(sl,52) = e2 = Ce(5,6) = Avg (s5,s6) = Avg(sl,55) ¨ Avg(sl,54) + s6 = e5-e4+e6 = etc.
As before, the proper combination of decoded frames can be precomputed and be available as a LUT.
[00048] An advantage of the proposed method is that it provides options for both content creators and users; i.e., in enables directorial/editorial choice and user choice. For example, preprocessing content in the encoder allows for a base frame rate to be created with various shutter angles. Each shutter angle can be assigned a temporal id value in the range [0, (n frames -1)], where n frames has a value equal to 120 divided by the base frame rate.
(For example, for a base frame rate of 24 fps, temporal id is in the range [0,4].) The choice may be made to optimize compression efficiency, or for aesthetic reasons. In some use cases, say, for over the top streaming, multiple bitstreams with different base layers can be encoded and stored and offered to users to select.
- 22 -[00049] In a second example of the disclosed methods, multiple backward compatible frame rates may be supported. Ideally, one may want to be able to decode at 24 frames per second to get a 24 fps base layer, at 30 frames per second to get a 30 fps sequence, at 60 frames per second to get a 60 fps sequence, and the like. If a target shutter angle is not specified, a default target shutter angle, among those shutter angles permissible for the source and target frame rates, as close as possible to 180 degrees is recommended. For example, for the values depicted in Table 7, preferred target shutter angles for fps at 120, 60, 40, 30, and 24 are 360, 180, 120, 180, and 216 degrees.
[00050] From the above examples it can be observed that the choice of how to encode the content can influence the complexity of decoding specific base layer frame rates. One embodiment of this invention is to adaptively choose the encoding scheme based on the desired base layer frame rate. For movie content this may be 24 fps, for example, while for sports it may be 60 fps.
[00051] Example syntax for the BC embodiment of the current invention is shown below and in Tables 8 and 9.
In SPS (Table 8), two syntax elements are added: sps hfr BC enabled flag, and sps base framerate (if sps hfr BC enabled flag is set equal to 1).
sps_hfr_BC_enabled_flag equal to 1 specifies that high frame rate with backward compatibility is enabled in the coded video sequence (CVS). sps hfr BC enabled flag equal to 0 specifies that high frame rate with backward compatibility is not enabled in the CVS.
sps base framerate specifies the base framerate for current CVS.
In tile group header, if sps hfr BC enabled flag is set to 1, the syntax number avg frames is .. sent in the bitstream.
number avg frames specifies the number of frames at the highest framerate (e.g., 120 fps) that are combined to generate the current picture at base framerate.
-23 -Table 8: Example RBSP syntax for input at various shutter angles seq_parameter_set_rbsp( ) {
Descriptor sps_max_sub_layers_minusl u(3) sps_reseryed_zero_5bits u(5) profile_tier_leveh sps_max_sub_layers_minusl ) sps_seq_parameter_set_id ue(v) ...
sps_hfr_BC_enabled_flag u (1) if( sps_hfr_BC_enabled_flag ) [ u (1) sps_base_frame_rate u (9) .1 ...
sps_extension_flag u(1) if( sps_extension_flag ) while( more_rbsp_data( ) ) sps_extension_data_flag u(1) rbsp_trailing_bits( )
- 24 -Table 9: Example picture parameter set RBSB syntax for input at various shutter angles pic_parameter_set_rbsp( ) {
Descriptor pps_pic_parameter_set_id ue(v) pps_seq_parameter_set_id ue(v) ...
if( sps_hfr_BC_enabled_flag ) number_avg_frames se(v) ===
rbsp_trailing_bits( ) Variations on the Second Embodiment (Fixed Frame Rate) [00052] The HEVC (H.265) coding standard (Ref.[1]) and the under development Versatile Video Coding Standard (commonly referred to as VVC, see Ref.[2]), define a syntax element, pic struct, that indicates whether a picture should be displayed as a frame or as one or more fields, and whether a decoded picture should be repeated. A copy of Table D.2, "Interpretation of pic struct," from HEVC is provided for ease of reference in the Appendix.
[00053] It is important to note that, as appreciated by the inventors, the existing pic struct syntax element can support only a specific subset of content frame rates when using a fixed frame rate coding container. For example, when using a fixed frame rate container of 60 fps, the existing pic struct syntax, when fixed pic rate within cvs flag is equal to 1, can support 30 fps by using frame doubling, and 24 fps by using frame doubling and frame tripling in alternating combination on every other frame. However, when using a fixed frame rate container of 120 fps, the current pic struct syntax cannot support frame rates of 24 fps nor 30 fps.
To alleviate this problem, two new methods are proposed: one is an extension of the HEVC
version, and the other is not.
- 25 -Method 1: pic_struct without backward compatibility [00054] VVC is still under development, thus one can design syntax with maximal freedom.
In an embodiment, in pic_struct, it is proposed to remove the options for frame doubling and frame tripling, use a specific value of pic_struct to indicate arbitrary frame repetition, and add a new syntax element, num frame repetition minus2, that specifies the number of frames to repeat. An example of the proposed syntax is described in the following Tables, where Table 10 denotes changes over Table D.2.3 in HEVC and Table 11 denotes changes of Table D.2 shown in the Appendix.
Table 10: Example picture timing SET message syntax, method 1 pic_timing( payloadSize ) 1 Descriptor if( frame_field_info_present_flag ) 1 pic_struct u(4) if( pic_struct == 7) u(4) num_frame_repetition_minus2 u(4) source_scan_type u(2) duplicate_flag u(1) .... (as the original) num frame repetition minus2 plus 2 indicates that when fixed pic rate within cvs flag is equal to 1, the frame should be displayed num frame repetition minus2 plus 2 times consectutively on displays with a frame refresh interval equal to .. DpbOutputElementalInterval[ n] as given by Equation E-73.
- 26 -Table 11: Example of revised of pic struct according to method 1 Value Indicated display of picture Restrictions 0 (progressive) Frame field_seq_flag shall be equal to 0 1 Top field field_seq_flag shall be equal to 1 2 Bottom field field_seq_flag shall be equal to 1 3 Top field, bottom field, in that field_seq_flag shall be equal to 0 order 4 Bottom field, top field, in that field_seq_flag shall be equal to 0 order Top field, bottom field, top field field_seq_flag shall be equal to 0 repeated, in that order 6 Bottom field, top field, bottom field_seq_flag shall be equal to 0 field repeated, in that order 7 Frame repetition field_seq_flag shall be equal to 0 fixed_pic_rate_within_cvs_flag shall be equal to 1 8 Top field paired with previous field_seq_flag shall be equal to 1 bottom field in output order 9 Bottom field paired with field_seq_flag shall be equal to 1 previous top field in output order Top field paired with next field_seq_flag shall be equal to 1 bottom field in output order 11 Bottom field paired with next field_seq_flag shall be equal to 1 top field in output order Method 2: Extended version of HEVC version of pic struct 5 [00055] AVC and HEVC decoders are already deployed, thus it may be desired to simply extend the existing pic struct syntax without removing old options. In an embodiment, a new
- 27 -pic_struct = 13, "frame repetition extension" value, and a new syntax element, num frame repetition minus4, are added. An example of the proposed syntax is described in Tables 12 and 13. For pic_struct values 0-12, the proposed syntax is identical with the one in Table D.2 (as shown in the Appendix), thus those values are omitted for simplicity.
Table 12: Example picture timing SET message syntax, method 2 pic_timing( payloadSize ) 1 Descriptor if( frame_field_info_present_flag ) 1 pic_struct u(4) if( pic_struct .13) u(4) num_frame_repetition_minus4 u(4) source_scan_type u(2) duplicate_flag u(1) ... (as the original) num frame repetition minus4 plus 4 indicates that when fixed pic rate within cvs flag is equal to 1, the frame should be displayed num frame repetition minus4 plus 4 times consectutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[ n] as given by Equation E-73.
Table 13: Example of revised pic_struct, method 2 Value Indicated display of picture Restrictions 0-12 As in Table D.2 As in Table D.2 13 Frame repetition extension field_seq_flag shall be equal to 0 fixed_pic_rate_within_cvs_flag shall be equal to 1 [00056] In HEVC, parameter frame field info present flag is present in the video usability information (VUI), but the syntax elements pic_struct, source scan type, and duplicate flag are
- 28 -in the pic timing() SET message. In an embodiment, it is proposed to move all related syntax elements to VUI, together with the frame field info present flag. An example of the proposed syntax is depicted in Table 14.
Table 14: Example VUI parameter syntax with support for the revised pic struct syntax element vui_parameters( ) {
Descriptor ... u(1) field_seq_flag u(1) frame_field_info_present_flag u(1) if( frame_field_info_present_flag ) {
pic_struct u(4) source _scan_type u(2) duplicate _flag u( 1 ) ]
= - =

Alternative signaling of shutter angle information [00057] When dealing with variable frame rate, it is desirable to identify both the desired frame rate and the desired shutter angle. In prior video coding standards, "Video Usability Information" (VUI) provides essential information for the proper display of video content, such as the aspect ratio, colour primaries, chroma sub-sampling, etc. VUI may also provide frame rate information if fixed pic rate is set to 1; however, there is no support for shutter angle information. Embodiments allow for different shutter angles to be used for different temporal layers, and a decoder can use shutter angle information to improve the final look on the display.
[00058] For example, HEVC supports temporal sub layers that essentially use frame dropping techniques to go from a higher frame rate to lower frame rate. The major problem with this is
- 29 -that the effective shutter angle is reduced with each frame drop. As an example, 60 fps can be derived from a 120 fps video by dropping every other frame; 30 fps can be derived by dropping 3 out of 4 frames; and 24 fps can be derived by dropping 4 out of 5 frames.
Assuming a full 360 degrees shutter for 120Hz, with simple frame dropping, the shutter angles for 60 fps, 30 fps, and 24 fps are 180, 90, and 72 degrees, respectively [3]. Experience has shown that shutter angles below 180 degrees are generally unacceptable, especially with frame rates below 50 Hz. By providing shutter angle information, for example, if it is desired that a display produces a cinematic effect from a 120 Hz video with reduced shutter angle for each temporal layer, smart techniques may be applied to improve the final look.
[00059] In another example, one may want to support a different temporal layer (say, a 60 fps sub-bitstream inside a 120 fps bitstream) with the same shutter angle. Then, the major problem is that when 120 fps video is displayed at 120Hz, the even/odd frames have different effective shutter angle. If a display has the related information, smart techniques can be applied to improve the final look. An example of the proposed syntax is shown in Table 15, where the .. E.2.1 VUI parameters syntax Table in HEVC (Ref. [1]) is modified to support shutter angle information as noted. Note that in another embodiment, instead of expressing shutter angle syntax in absolute degrees, it can alternatively be expressed as ratio of frame rate over shutter speed (see equation (1)).
-30-Table 15: Example VUI parameter syntax with shutter angle support vui_parameters( ) 1 Descriptor ...
vuLtiming_info_present_flag u(1) if( vui_timing_info_present_flag ) 1 vui_num_units_in_tick u(32) vui_time_scale u(32) vui_poc_proportional_to_timing_flag u(1) if( vui_poc_proportional_to_timing_flag ) vui_num_ticks_poc_diff one_minusl ue(v) vui_hrd_parameters_present_flag u(1) if( vui_hrd_parameters_present_flag ) hrd_parameters( 1, sps_max_sub_layers_minus1 ) vui_shutter_angle_info_present_flag u(1) if( vui_shutter_angles_info_present_flag ) {
fixed_shutter_angle_within_cvs _flag u(1) if (fixed_shutter_angle_with_cvs_flag ) fixed_shutter_angle u(9) else {
for( i = 0; i <= sps_max_sub_layers_minusl; i++ ) I
sub_layer_shutter_angle[ i] u(9) .1 .1 ...
-31-vui shutter angle info present flag equal to 1 specifies that shutter angle information is present in the vui parameters( ) syntax structure. vui shutter angle info present flag equal to 0 specifies that shutter angle information is not present in the vui parameters( ) syntax structure.
fixed shutter angle within cvs flag equal to 1 specifies that shutter angle information is the same for all temporal sub-layers in the CVS. fixed shutter angle within cvs flag equal to 0 specifies that shutter angle information may not be the same for all temporal sub-layers in the CVS.
fixed shutter angle specifies the shutter angle in degrees within a CVS. The value of fixed shutter angle shall be in the range of 0 to 360.
sub layer shutter angle[ i ] specifies the shutter angle in degrees when HighestTid is equal to i. The value of sub layer shutter angle[ i ] shall be in the range of 0 to 360.
Gradual frame-rate update within a coded video sequence (CVS) [00060] Experiments have shown that for HDR content displayed on an HDR
display, to perceive the same motion juddering as standard dynamic range (SDR) playback in a 100 nits display, the frame rate needs to be increased based on the brightness of the content. In most standards (AVC, HEVC, VVC, etc.), the video frame rate can be indicated in the VUI (contained in SPS) using the vui time scale, vui num units in tick and elemental duration in tc minusl[temporal id max] syntax elements, for example, as shown in Table 16 below (see Section E.2.1 in Ref.[1]).
- 32-Table 16: VUI syntax elements to indicate frame rate in HEVC
vui_parameters( ) {
Descriptor ...
vui_timing_info_present_flag u(1) if( vui_timing_info_present_flag ) {
vui_num_units_in_tick u(32) vuLtime_scale u(32) vui_poc_proportional_to_timing_flag u(1) if( vui_poc_proportional_to_timing_flag ) vui num ticks poc diff one minusl ue(v) vui_hrd_parameters_present_flag u(1) if( vui_hrd_parameters_present_flag ) hrd_parameters( 1, sps_max_sub_layers_minusl ) ....
As discussed in Ref. [1], The variable ClockTick is derived as follows and is called a clock tick:
ClockTick ¨ vui num units in tick vui time scale ¨ _ _ _ _ . _ _ picture_duration = ClockTick * ( Iii]elemental_duration_in_tc_minusl + 1) frame_rate = 1/pic_duration.
[00061] However, the frame rate can only be changed at specific time instants, for example, in HEVC, only at intra random access point (TRAP) frames or at the start of a new CVS. For HDR playback, when there is a fade-in or fade-out case, because the brightness of a picture is changing frame by frame, there might be a need to change frame rate or picture duration for every picture. To allow frame rate or picture duration refresh at any time instant (even on a frame-by-frame basis), in an embodiment, a new SET message for "gradual refresh rate" is proposed, as shown in Table 17.
- 33 -Table 17:Example syntax to support gradual refresh frame rate in SET messaging gradual_refresh_rate( payloadSize ) 1 Descriptor num_units_in_tick u(32) time_scale u(32) [00062] The definition of new syntax num units in tick is the same as vui num units in tick, and the definition of time scale is the same as that of vui time scale.
num units in tick is the number of time units of a clock operating at the frequency time scale Hz that corresponds to one increment (called a clock tick) of a clock tick counter.
num units in tick shall be greater than 0. A clock tick, in units of seconds, is equal to the quotient of num units in tick divided by time scale. For example, when the picture rate of a video signal is 25 Hz, time scale may be equal to 27 000 000 and num units in tick may be equal to 1 080 000 and consequently a clock tick may be equal to 0.04 seconds.
time scale is the number of time units that pass in one second. For example, a time coordinate system that measures time using a 27 MHz clock has a time scale of 27 000 000.
The value of time scale shall be greater than 0.
The picture duration time for the picture which uses gradual refresh rate SET
message is defined as:
picture duration = num units in tick time scale.
Signalling of Shutter Angle Information via SET Messaging [00063] As discussed earlier, Table 15 provides an example of VUI parameter syntax with shutter angle support. As an example, and without limitation, Table 18 lists identical syntax elements, but now as part of an SET message for shutter angle information.
Note that SET
messaging is being used only as an example and similar messaging may be constructed at other
- 34-layers of high-level syntax, such as the Sequence Parameter Set (SPS), the Picture Parameter Set (PPS), the Slice or Tile Group header, and the like.
Table 18: Example SET message syntax for shutter angle information shutter_angle_information ( payloadSize ) 1 Descriptor fixed_shutter_angle_within_cvs_flag u(1) if (fixed_shutter_angle_within_cvs_flag) fixed_shutter_angle u(9) else I
for( i = 0; i <= sps_max_sub_layers_minusl; i++) sub_layer_shutter_angle] i] u(9) I

[00064] Shutter angle is typically expressed in degrees from 0 to 360 degrees.
For example, a shutter angle of 180 degrees indicates that the exposure duration is 1/2 the frame duration. Shutter angle may be expressed as: shutter angle = frame rate * 360*shutter speed, where shutter speed is the exposure duration and frame rate is the inverse of frame duration.
frame rate for the given temporal sub-layer Tid may be indicated by the num units in tick, time scale , elemental duration in tc minusl[Tid]. For example, when fixed pic rate within cvs flag[ Tid ] is equal to 1:
frame rate = time scale / ( num units in tick * (elemental duration in tc minusl[Tid] + 1 ) ) =
[00065] In some embodiments, the value of shutter angle (e.g., fixed shutter angle) may not be an integer, for example, it may be 135.75 degrees. To allow more precision, in Table 21, one may replace u(9) (unsigned 9-bits) with u(16) or some other suitable bit-depth (e.g., 12 bits, 14 bits, or more than 16 bits).
[00066] In some embodiments, it may be beneficial to express shutter angle information in terms of "Clock ticks." In VVC, the variable ClockTick is derived as follows:
- 35 -ClockTick = num units in tick time scale .
(8) Then, one can express both frame duration and exposure duration as multiple or fractional of clock ticks:
exposure duration = fN*ClockTick , (9) frame duration = fM*ClockTick , (10) where fN and fM are floating-point values and fN < fM.
Then shutter angle = frame rate * 360*shutter speed =
= (1/frame duration)*360*exposure duration =
(11) =(exposure duration*360)/frame duration =
=(f1.4*ClockTick*360)/(fM*ClockTick ) =
= (fN/fM ) * 360 = (Numerator/Denominator) * 360, where Numerator and Denominator are integers approximating the fN/fM ratio.
[00067] Table 19 shows an example of SET messaging indicated by equation (11).
In this example, shutter angle must be larger than 0 for a real-world camera.
- 36 -Table 19: Example SET messaging for shutter angle information based on clock ticks shutter_angle_information ( payloadSize ) { Descriptor fixed_shutter_angle_within_cys_flag u(1) if (fixed_shutter_angle_within_cvs_flag) {
fixed_shutter_angle_numer_minusl u(16) fixed_shutter_angle_denom_minusl u(16) else {
for( i = 0; i <= sps_max_sub_layers_minusl; i++) {
sub_layer_shutter_angle_numer_minusl Iii]
u(16) sub_layer_shutter_angle_denom_minusl Iii]
u(16) I

As discussed earlier, the use of u(16) (unsigned 16 bits) for shutter angle precision is depicted as an example and corresponds to a precision of: 360/216 = 0.0055. The precision can be adjusted based on real applications. For example, using u(8), the precision is 360/28 =
1.4063.
NOTE ¨ Shutter angle is expressed in degrees greater than 0 but less than or equal to 360 degrees. For example, a shutter angle of 180 degrees indicates that the exposure duration is 1/2 the frame duration.
fixed shutter angle within cvs flag equal to 1 specifies that shutter angle value is the same for all temporal sub-layers in the CVS. fixed shutter angle within cvs flag equal to 0 specifies that shutter angle value may not be the same for all temporal sub-layers in the CVS.
fixed shutter angle numer minusl plus 1 specifies the numerator used to derive shutter angle value. The value of fixed shutter angle numer minusl shall be in the range of 0 to 65535, inclusive.
fixed shutter angle demom minusl plus 1 specifies the denominator used to derive shutter angle value. The value of fixed shutter angle demom minusl shall be in the range of 0 to 65535, inclusive.
- 37 -The value of fixed shutter angle numer minusl shall be less than or equal to the value of fixed shutter angle demom minusl.
The variable shutterAngle in degree is derived as follows:
shutterAngle = 360 * (fixed shutter angle numer minus 1 +
1) (fixed shutter angle demom minusl + 1)) sub layer shutter angle numer minusil i ] plus 1 specifies the numerator used to derive shutter angle value when HighestTid is equal to i. The value of sub layer shutter angle numer minusl[ i ] shall be in the range of 0 to 65535, inclusive.
sub layer shutter angle demom minusl [ i ] plus 1 specifies the denominator used to derive shutter angle value when HighestTid is equal to i. The value of sub layer shutter angle demom minusl[ i] shall be in the range of 0 to 65535, inclusive.
The value of sub layer shutter angle numer minusl[ i ] shall be less than or equal to the value of sub layer shutter angle denom minusl[ i].
The variable subLayerShutterAngle[ i] in degree is derived as follows:
subLayerShutterAngle[ ii = 360 *
(sub layer shutter angle numer minusl[ i ] + 1) (sub layer shutter angle demom minusl[ i ] + 1) [00068] In another embodiment, frame duration (e.g., frame duration) may be specified by some other means. For example, in DVB/ATSC, when fixed pic rate within cvs flag[ Tid ] is equal to 1:
frame rate = time scale / ( num units in tick * (elemental duration in tc minusl[Tid] + 1 ) ), frame duration = 1 / frame rate.
[00069] The syntax in Table 19 and in some of the subsequent Tables assumes that the shutter angle will always be greater than zero; however, shutter angle = 0 can be used to signal a creative intent where the content should be displayed without any motion blur.
Such could be the case for moving graphics, animation, CGI textures and mat screens, etc. As such, for
- 38 -example, signalling shutter angle = 0 could be useful for mode decision in a transcoder (e.g., to select transcoding modes that preserve edges) as well as in a display that receives the shutter angle metadata over a CTA interface or 3GPP interface. For example, shutter angle = 0 could be used to indicate to a display that is should not perform any motion processing such as denoising, frame interpolation, and the like. In such an embodiment, syntax elements fixed shutter angle numer minusl and sub layer shutter angle numer minusl[ i ]
may be replaced by the syntax elements fixed shutter angle numer and sub layer shutter angle numer[ i ], where fixed shutter angle numer specifies the numerator used to derive shutter angle value. The value of fixed shutter angle numer shall be in the range of 0 to 65535, inclusive.
sub layer shutter angle numer[ ii specifies the numerator used to derive shutter angle value when HighestTid is equal to i. The value of sub layer shutter angle numer[ ii shall be in the range of 0 to 65535, inclusive.
[00070] In another embodiment, fixed shutter angle denom minusl and sub layer shutter angle denom minusl[ i ] can also be replaced by the syntax elements fixed shutter angle denom and sub layer shutter angle denom[ i ] as well.
[00071] In an embodiment, as depicted in Table 20, one can reuse the num units in tick and time scale syntax defined in SPS by setting general hrd parameters present flag equal to 1 in VVC. Under this scenario, the SET message can be renamed as Exposure Duration SET message.
- 39 -Table 20: Example SET messaging for signaling exposure duration exposure_duration_information ( payloadSize ) { Descriptor fixed_exposure_duration_within_cvs_flag u(1) if (fixed_shutter_angle_within_cvs_flag) {
fixed_exposure_duration_numer_minusl u(16) fixed_exposure_duration_denom_minusl u(16) else {
for( i = 0; i <= sps_max_sub_layers_minusl; i++) {
sub_layer_exposure_duration_numer_minusl Iii]
u(16) sub_layer_exposure_duration_denom_minusl Iii]
u(16) I
I
fixed exposure duration within cvs flag equal to 1 specifies that effecitive exposure duration value is the same for all temporal sub-layers in the CVS.
fixed exposure duration within cvs flag equal to 0 specifies that effective exposure duration value may not be the same for all temporal sub-layers in the CVS.
fixed exposure duration numer minusl plus 1 specifies the numerator used to derive exposure duration value. The value of fixed exposure duration numer minusl shall be in the range of 0 to 65535, inclusive.
fixed exposure duration demom minusl plus 1 specifies the denominator used to derive exposure duration value. The value of fixed exposure duration demom minusl shall be in the range of 0 to 65535, inclusive.
The value of fixed exposure during numer minusl shall be less than or equal to the value of fixed exposure duration demom minusl.
- 40 -The variable fixedExposureDuration is derived as follows:
fixedExposureDuration = ( fixed exposure duration numer minusl + 1 ) ( fixed exposure duration demom minus 1 + 1) * ClockTicks sub layer exposure duration numer minusl[ i ] plus 1 specifies the numerator used to derive exposure duration value when HighestTid is equal to i. The value of sub layer exposure duration numer minusl[ i ] shall be in the range of 0 to 65535, inclusive.
sub layer exposure duration demom minusl [ i ] plus 1 specifies the denominator used to derive exposure duration value when HighestTid is equal to i. The value of sub layer exposure duration demom minusl[ i ] shall be in the range of 0 to 65535, inclusive.
The value of sub layer exposure duration numer minusl[ i ] shall be less than or equal to the value of sub layer exposure duration demom minusl[ i ].
The variable subLayerExposureDuration[ i ] for HigestTid equal to i is derived as follows:
subLayerExposureDuration[ i ] = ( sub layer exposure duration numer minusl[ i ] + 1 ) ( sub layer exposure duration demom minusl[ i ] + 1) * ClockTicks.
[00072] In another embodiment, as shown in Table 21, one may explicitly define clockTick by the syntax elements expo num units in tick and expo time scale. The advantage here is that it does not rely on whether general hrd parameters present flag set equal to 1 in VVC as the previous embodiment, then clockTick = expo num units in tick expo time scale . (12)
- 41 -Table 21: Example SET messaging for exposure time signaling exposure_duration_information ( payloadSize ) {
Descriptor expo_num_units_in_tick u(32) expo_time_scale u(32) fixed_exposure_duration_within_cvs_flag u(1) if (!fixed_exposure_duration_within_cvs_flag) for( i = 0; i <= sps_max_sub_layers_minus1; i++) {
sub_layer_exposure_duration_numer_minusl Iii]
u(16) sub_layer_exposure_duration_denom_minusl Iii]
u(16) expo num units in tick is the number of time units of a clock operating at the frequency time scale Hz that corresponds to one increment (called a clock tick) of a clock tick counter.
expo num units in tick shall be greater than 0. A clock tick, defined by variable clockTick, in units of seconds, is equal to the quotient of expo num units in tick divided by expo time scale.
expo time scale is the number of time units that pass in one second.
clockTick = expo num units in tick expo time scale.
NOTE: The two syntax elements: expo num units in tick and expo time scale are defined to measure exposure duration.
It is a requirement for bitstream conformance that clockTick shall be less than or equal to ClockTick when num units in tick and time scale are present.
fixed exposure duration within cvs flag equal to 1 specifies that effecitive exposure duration value is the same for all temporal sub-layers in the CVS.
fixed exposure duration within cvs flag equal to 0 specifies that effecitive exposure duratiton value may not be the same for all temporal sub-layers in the CVS. When
- 42 -fixed exposure duration within cvs flag equal to 1, the variable fixedExposureDuration is set equal to clockTick.
sub layer exposure duration numer minusl[ i ] plus 1 specifies the numerator used to derive exposure duration value when HighestTid is equal to i. The value of sub layer exposure duration numer minusl[ i] shall be in the range of 0 to 65535, inclusive.
sub layer exposure duration demom minusl [ i] plus 1 specifies the denominator used to derive exposure duration value when HighestTid is equal to i. The value of sub layer exposure duration demom minusl[ i ] shall be in the range of 0 to 65535, inclusive.
The value of sub layer exposure duration numer minusl[ i ] shall be less than or equal to the value of sub layer exposure duration demom minusl[ i].
The variable subLayerExposureDuration[ ii for HigestTid equal to i is deruved as follows:
subLayerExposureDuration[ i] = ( sub layer exposure duration numer minusl[ i ]
+ 1 ) ( sub layer exposure duration denom minusl[ i ] + 1) * clockTick.
[00073] As discussed earlier, syntax parameters sub layer exposure duration numer minusl[ i ] and sub layer exposure duration denom minusl[ i ] may also be replaced by sub layer exposure duration numer[ i] and sub layer exposure duration denom[
i].
[00074] In another embodiment, as shown in Table 22, one may define the parameter ShutterInterval (i.e., exposure duration) by the syntax elements sii num units in shutter interval and sii time scale, where ShutterInterval = sii num units in shutter interval sii time scale .
(13)
-43 -Table 22: Example SET messaging for exposure duration (shutter interval information) signaling shutter_interval_information ( payloadSize ) {
Descriptor sii_num_units_in_shutter_interval u(32) sii_time_scale u(32) fixed_shutter_interval_within_cvs_flag u(1) if ( !fixed_shutter_interval_within_cvs_flag ) for( i = 0; i <= sps_max_sub_layers_minus1; i++) {
sub_layer_shutter_interval_numed i] u(16) Iii]sub_layer_shutter_interval_denom u(16) Shutter interval information SET message semantics .. The shutter interval information SET message indicates the shutter interval for the associated video content prior to encoding and display ¨ e.g., for camera-captured content, the amount of time that an image sensor was exposed to produce a picture.
sii num units in shutter interval specifies the number of time units of a clock operating at the frequency sii time scale Hz that corresponds to one increment of an shutter clock tick counter. Shutter interval, defined by variable ShutterInterval, in units of seconds, is equal to the quotient of sii num units in shutter interval divided by sii time scale. For example, when ShutterInterval is equal to 0.04 seconds, sii time scale may be equal to 27 000 000 and sii num units in shutter interval may be equal to 1 080 000.
sii time scale specifies the number of time units that pass in one second. For example, a time .. coordinate system that measures time using a 27 MHz clock has a sii time scale of 27 000 000.
When the value of sii time scale is greater than 0, the value of ShutterInterval is specified by:
ShutterInterval = sii num units in shutter interval sii time scale Otherwise (the value of sii time scale is equal to 0), ShutterInterval should be interpreted as unknown or unspecified.
- 44 -NOTE 1 ¨ A value of ShutterInterval equal to 0 may indicate that the associated video content contains screen capture content, computer generated content, or other non-camera-capture content.
NOTE 2 ¨ A value of ShutterInterval greater than the value of the inverse of the coded picture rate, the coded picture interval, may indicate that the coded picture rate is greater than the picture rate at which the associated video content was created ¨ e.g., when the coded picture rate is 120 Hz and the picture rate of the associated video content prior to encoding and display is 60 Hz. The coded interval for the given temporal sub-layer Tid may be indicated by ClockTick and elemental duration in tc minusl[ Tid]. For example, when fixed pic rate within cvs flag[ Tid] is equal to 1, picture interval for the given temporal sub-layer Tid, defined by variable PictureInterval[ Tid ], may be specified by:
PictureInterval[ Tid] = ClockTick * ( elemental duration in tc minusl[ Tid] +
1 ) .
fixed shutter interval within cvs flag equal to 1 specifies that the value of ShutterInterval is the same for all temporal sub-layers in the CVS. fixed shutter interval within cvs flag equal to 0 specifies that value of ShutterInterval may not be the same for all temporal sub-layers in the CVS.
sub layer shutter interval numer[ i ] specifies the numerator used to derive sub layer shutter interval, defined by variable subLayerShutterInterval[ i I, in units of seconds, when HighestTid is equal to i.
sub layer shutter interval denom[ i ] specifies the denominator used to derive sub layer shutter interval, defined by variable subLayerShutterInterval[ i I, in units of seconds, when HighestTid is equal to i.
The value of subLayerShutterInterval[ i I for HighestTid equal to i is derived as follows. When the value of fixed shutter interval within cvs flag is equal to 0 and the value of sub layer shutter interval denom[ i I is greater than 0:
subLayerShutterInterval[ i I = ShutterInterval * sub layer shutter interval numer[ i I
sub layer shutter interval denom[ i I
Otherwise (the value of sub layer shutter interval denom[ i I is equal to 0), subLayerShutterInterval[ i I should be interpreted as unkown or unspecified.
When the value of fixed shutter interval within cvs flag is not equal to 0, subLayerShutterInterval[ i I = ShutterInterval.
- 45 -[00075] In an alternative embodiment, instead of using a numerator and a denominator for signaling the sub-layer shutter interval, one uses a single value. An example of such syntax is shown in Table 23.
Table 23: Example SET messaging for shutter interval signaling shutter_interval_information ( payloadSize ) {
Descriptor sii_num_units_in_shutter_interval u(32) sii_time_scale u(32) fixed_shutter_interval_within_cvs_flag u(1) if ( !fixed_shutter_interval_within_cvs_flag ) for( i = 0; i <= sps_max_sub_layers_minus1; i++) {
sub_layer_num_units_in_shutter_intervah i]
u(32) Shutter interval information SET message semantics The shutter interval information SET message indicates the shutter interval for the associated video content prior to encoding and display ¨ e.g., for camera-captured content, the amount of time that an image sensor was exposed to produce a picture.
sii num units in shutter specifies the number of time units of a clock operating at the frequency sii time scale Hz that corresponds to one increment of an shutter clock tick counter.
Shutter interval, defined by variable ShutterInterval, in units of seconds, is equal to the quotient of sii num units in shutter interval divided by sii time scale. For example, when ShutterInterval is equal to 0.04 seconds, sii time scale may be equal to 27 000 000 and sii num units in shutter interval may be equal to 1 080 000.
sii time scale specifies the number of time units that pass in one second. For example, a time coordinate system that measures time using a 27 MHz clock has a sii time scale of 27 000 000.
When the value of sii time scale is greater than 0, the value of ShutterInterval is specified by:
ShutterInterval = sii num units in shutter interval sii time scale Otherwise (the value of sii time scale is equal to 0), ShutterInterval should be interpreted as unknown or unspecified.
- 46 -NOTE 1 ¨ A value of ShutterInterval equal to 0 may indicate that the associated video content contain screen capture content, computer generated content, or other non-camera-capture content.
NOTE 2 ¨ A value of ShutterInterval greater than the value of the inverse of the coded picture rate, the coded picture interval, may indicate that the coded picture rate is greater than the picture rate at which the associated video content was created ¨ e.g., when the coded picture rate is 120 Hz and the picture rate of the associated video content prior to encoding and display is 60 Hz. The coded picture interval for the given temporal sub-layer Tid may be indicated by ClockTick and elemental duration in tc minusl[ Tid]. For example, when fixed pic rate within cvs flag[ Tid] is equal to 1, picture interval for the given temporal sub-layer Tid, defined by variable PictureInterval[ Tid ], may be specified by:
PictureInterval[ Tid] = ClockTick * ( elemental duration in tc minusl[ Tid] +
1 ) .
fixed shutter interval within cvs flag equal to 1 specifies that the value of ShutterInterval is the same for all temporal sub-layers in the CVS. fixed shutter interval within cvs flag equal to 0 specifies that value of ShutterInterval may not be the same for all temporal sub-layers in the CVS.
sub layer num units in shutter interval[ i ] specifies the number of time units of a clock operating at the frequency sii time scale Hz that corresponds to one increment of an shutter clock tick counter. Sub layer shutter interval, defined by variable subLayerShutterInterval[ i I, in units of seconds, when HighestTid is equal to i, is equal to the quotient of sub layer num units in shutter interval[ i I divided by sii time scale.
When the value of fixed shutter interval within cvs flag is equal to 0 and the value of sii time scale is greater than 0, the value of subLayerShutterInterval[ i I is specified by:
subLayerShutterInterval[ i I = sub layer num units in shutter interval[ i I
sii time scale Otherwise (the value of sii time scale is equal to 0), subLayerShutterInterval[ i I should be interpreted as unknown or unspecified. When the value of fixed shutter interval within cvs flag is not equal to 0, subLayerShutterInterval[ i I = ShutterInterval.
- 47 -[00076] Table 24 provides a summary of the six approaches discussed in Tables 18-23 for providing SET messaging related to shutter angle or exposure duration.
Table 24: Summary of SET messaging approaches for signaling signal shutter angle information Table No. Key signaling elements and dependencies 18 Shutter angle (0 to 360) is signaled explicitly 19 Shutter angle is expressed as a ratio of Numerator and Denominator values to be scaled by 360 (the clock-tick value is implied) 20 Exposure duration is signaled as a ratio of Numerator and Denominator values (the clock-tick value is implied) 21 Exposure duration is signaled as a ratio of Numerator and Denominator values; the clock tick-value is signaled explicitly as a ratio of two values 22 Shutter interval information is signaled as the ratio of two values: the number of clock-tick units in the exposure and an exposure-time scale;
Sub-layer-related exposure times are signaled as a ratio of two values 23 Shutter interval information or exposure duration is signaled as the ratio of two values: the number of clock-tick units in the exposure and an exposure-time scale; Sub-layer-related exposure times are signaled as the number of clock-tick units in the exposure in each sub-layer Variable frame rate signalling [00077] As discussed in U.S. Provisional Application 62/883,195, filed on Aug. 06, 2019, in many applications it is desired for a decoder to support playback at variable frame rates. Frame rate adaptation is typically part of the operations in the hypothetical reference decoder (HRD), as described, for example, in Annex C of Ref. [2]. In an embodiment, it is proposed to signal via SET messaging or other means a syntax element defining picture presentation time (PPT) as function of a 90 kHz clock. This is kind of repetition of the nominal decoder picture buffer (DPB) output time as specified in the HRD, but now using a 90 kHz ClockTicks precision as specified in the MPEG-2 system. The benefit of this SET message are a) if HRD
is not enabled,
- 48 -one can still use the PPT SET message to indicate timing for each frame; b) it can ease the translation of bitstream timing and system timing.
[00078] Table 25 describes an example of the syntax of the proposed PPT timing message, which matches the syntax of the presentation time stamp (PTS) variable being used in MPEG-2 transport (H.222) (Ref.[4]).
Table 25: Example syntax for picture presentation time messaging picture_presentation_time ( payloadSize ) {
Descriptor PPT u(33) PPT (picture presentation time) Presentation times shall be related to decoding times as follows: The PPT is a 33-bit number coded in three separate fields. It indicates the time of presentation, tpn(k), in the system target decoder of a presentation unit k of elementary stream n. The value of PPT is specified in units of the period of the system clock frequency divided by 300 (yielding 90 kHz). The picture .. presentation time is derived from the PPT according to equation below.
PPT(k) = ((system clockjrequency x tmi(k))/300)%233 where tpn(k) is the presentation time of presentation unit P.(k).
Shutter Interval Messaging in AVC
[00079] In an embodiment, if a shutter interval information (SIT) SET message exists for any picture in a coded video sequence (CVS), then it is suggested that it must exist in the first access unit of the CVS. Unlike HEVC, a temporal index (which is used to identify a sub-layer index) .. does not exist in an AVC single-layer bitstream. To address this issue when the shutter interval is not fixed within a CVS, it is proposed that a shutter interval information SET
message shall be present for every picture to assign a value for sii sub layer idx to each picture to identify the
- 49 -sub-layer index of the current picture. Other shutter interval related information shall be presented only for the first access unit of the CVS and persist until a new CVS begins or the bitstream ends.
[00080] In AVC, an access unit is defined as a set of NAL units that are consecutive in .. decoding order and contain exactly one primary coded picture. In addition to the primary coded picture, an access unit may also contain one or more redundant coded pictures, one auxiliary coded picture, or other NAL units not containing slices or slice data partitions of a coded picture.
The decoding of an access unit always results in a decoded picture.
[00081] Example syntax element values for the case in which shutter interval is fixed for the CVS is shown in Table 26. Example syntax element values for the first and subsequent shutter interval information SET message for the case in which shutter interval may be different for different sub-layers is shown in Table 27. In Tables 26 and 27, cells with "(none)" indicate that no value is signalled in the shutter interval information SET message for the corresponding syntax element.
Table 26: Example of shutter interval information SET message syntax element values for fixed shutter interval for IDR access unit 1st shutter interval info syntax element SEI message in the CVS
sii_sub_layer_idx 0 shutter_interval_info_present_flag 1 sii_time_scale u(32) fixed_shutter_interval_within_cvs_flag 1 sii_num_units_in_shutter_interval u(32) sii_max_sub_layers_minusl (none) sub_layer_num_units_in_shutter_intervall i (none)
- 50 -Table 27: Example of shutter interval information SET message syntax element values for non-fixed shutter interval for IDR access unit 1st shutter interval info Subsequent shutter interval syntax element SEI message info SEI messages in the CVS
in the CVS
sii_sub_layer_idx 0 0 ue(v) > 0 shutter_interval_info_present_flag 1 0 (none) sii_time_scale u(32) (none) (none) fixed_shutter_interval_within_cvs_flag 0 (none) (none) sii_num_units_in_shutter_interval (none) (none) (none) sii_max_sub_layers_minusl u(3) (none) (none) sub_layer_num_units_in_shutter_intervalli ii u(32) (none) (none) [00082] Table 28 depicts an example syntax structure for SIT SET messaging in AVC.
- 51 -Table 28: Example SIT SET message syntax in AVC
shutter_interval_info( payloadSize ) { C Descriptor sii_sub_layer_idx 5 ue(v) if( sii_sub_layer_idx = = 0) shutter_interyal_info_present_flag 5 u(1) if( shutter_interval_info_present_flag ) sii_time_scale 5 u(32) fixed_shutter_interyal_within_cys_flag 5 u(1) if( fixed_shutter_interval_within_cvs_flag ) sii_num_units_in_shutter_interyal 5 u(32) else {
sii_max_sub_layers_minusl 5 u(3) for( i = 0; i <= sii_max_sub_layers_minusl; i++) sub_layer_num_units_in_shutter_interyall i] 5 u(32) [00083] The shutter interval information SET message indicates the shutter interval for the associated video source pictures prior to encoding and display, e.g., for camera-captured content, the shutter interval is amount of time that an image sensor is exposed to produce each source picture.
sii_sub_layer_idx specifies the shutter interval temporal sub-layer index of the current picture.
The value of sii_sub_layer_idx shall be equal to 0 when the current access unit is the first access unit of the CVS. When fixed shutter interval within cvs flag is equal to 1, the value of sii_sub_layer_idx shall be equal to 0. Otherwise, fixed shutter interval within cvs flag is equal to 0, the value of sii_sub_layer_idx shall be less than or equal to the value of sii max sub layers minusl.
- 52 -shutter interval info present flag equal to 1 indicates that the syntax elements sii time scale, fixed shutter interval within cvs flag, and either sii num units in shutter interval or sii max sub layers minusl and sub layer num units in shutter interval[ i I are present.
shutter interval info present flag equal to 0 indicates that the syntax elements sii time scale, fixed shutter interval within cvs flag, sii num units in shutter interval, sii max sub layers minusl, and sub layer num units in shutter interval[ i I
are not present.
The value of shutter interval info present flag shall be equal to 1 when the current access unit is the first access unit of the CVS. Otherwise, the current access unit is not the first access unit of the CVS, the value of shutter interval info present flag shall be equal to 0.
sii time scale specifies the number of time units that pass in one second. The value of sii time scale shall be greater than 0. For example, a time coordinate system that measures time using a 27 MHz clock has an sii time scale of 27 000 000.
fixed shutter interval within cvs flag equal to 1 specifies that the indicated shutter interval is the same for all pictures in the CVS. fixed shutter interval within cvs flag equal to 0 specifies that the indicated shutter interval may not be the same for all pictures in the CVS.
sii num units in shutter interval, when fixed shutter interval within cvs flag is equal to 1, specifies the number of time units of a clock operating at the frequency sii time scale Hz that corresponds to the indicated shutter interval of each picture in the CVS. The value 0 may be used to indicate that the associated video content contains screen capture content, computer generated content, or other non-camera-captured content.
The indicated shutter interval, denoted by the variable shutterInterval, in units of seconds, is equal to the quotient of sii num units in shutter interval divided by sii time scale. For example, to represent a shutter interval equal to 0.04 seconds, sii time scale may be equal to 27 000 000 and sii num units in shutter interval may be equal to 1 080 000.
sii max sub layers minusl plus 1 specifies the maximum number of shutter interval temporal sub-layers indexes that may be present in the CVS.
sub layer num units in shutter interval[ i], when present, specifies the number of time units of a clock operating at the frequency sii time scale Hz that corresponds to the shutter interval of each picture in the CVS for which the value of sii sub layer idx is equal to i. The sub-layer shutter interval for each picture for which the value of sii sub layer idx is equal to i,
- 53 -denoted by the variable subLayerShutterInterval[ i ], in units of seconds, is equal to the quotient of sub layer num units in shutter interval[ i ] divided by sii time scale.
The variable subLayerShutterInterval[ i ], corresponding to the indicated shutter interval of each picture in the sub-layer representation with Temporand equal to i in the CVS, is thus derived as follows:
if( fixed shutter interval within cvs flag ) subLayerShutterInterval[ i ] = sii num units in shutter interval sii time scale else subLayerShutterInterval[ i ] = sub layer num units in shutter interval[ i ]
sii time scale When a shutter interval information SET message is present for any access unit in a CVS, a shutter interval information SET message shall be present for the IDR access unit that is the first access unit of the CVS. All shutter interval information SET messages that apply to the same access unit shall have the same content.
sii time scale and fixed shutter interval within cvs flag persist from the first access unit of the CVS until a new CVS begins or the bitstream ends.
When the value of fixed shutter interval within cvs flag is equal to 0, a shutter interval information SET message shall be present for every picture in the CVS. When present, sii num units in shutter interval, sii max sub layers minusl, and sub layer num units in shutter interval[ i ], persist from the first access unit of the CVS until a new CVS begins or the bitstream ends.
References Each one of the references listed herein is incorporated by reference in its entirety.
[1] High efficiency video coding, H.265, Series H, Coding of moving video, ITU, (02/2018).
[2] B. Bross, J. Chen, and S. Liu, "Versatile Video Coding (Draft 5)," JVET
output document, JVET-N1001, v5, uploaded May 14, 2019.
- 54 -[3] C. Carbonara, J. DeFilippis, M. Korpi, "High Frame Rate Capture and Production," SMPTE
2015 Annual Technical Conference and Exhibition, Oct 26-29, 2015.
[4] Infrastructure of audiovisual services ¨ Transmission multiplexing and synchronization, H.222.0, Series H, Generic coding of moving pictures and associated audio information:
Systems, ITU, 08/2018.
EXAMPLE COMPUTER SYSTEM IMPLEMENTATION
[00084] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components. The computer and/or IC may perform, control, or execute instructions relating to frame-rate scalability, such as those described herein. The computer and/or IC may compute any of a variety of parameters or values that relate to frame-rate scalability described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.
[00085] Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention.
For example, one or more processors in a display, an encoder, a set top box, a transcoder or the like may implement methods related to frame-rate scalability as described above by executing software instructions in a program memory accessible to the processors.
Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non-transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.
- 55 -Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.
EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS
[00086] Example embodiments that relate to frame-rate scalability are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation.
Thus, the sole and exclusive indicator of what is the invention and what is intended by the applicants to be the invention is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly .. set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
- 56 -Appendix This Appendix provides a copy of Table D.2 and associated pic struct-related information from the H.265 specification (Ref. [1]).
Table D.2 ¨ Interpretation of pic struct Value Indicated display of picture Restrictions 0 (progressive) Frame field seq flag shall be equal to 0 1 Top field field seq flag shall be equal to 1 2 Bottom field field seq flag shall be equal to 1 3 Top field, bottom field, in field seq flag shall be equal to 0 that order 4 Bottom field, top field, in that field seq flag shall be equal to 0 order Top field, bottom field, top field seq flag shall be equal to 0 field repeated, in that order 6 Bottom field, top field, field seq flag shall be equal to bottom field repeated, in that order 7 Frame doubling field seq flag shall be equal to 0 fixed pic rate within cvs flag shall be equal to 1 8 Frame tripling field seq flag shall be equal to 0 fixed pic rate within cvs flag shall be equal to 1 9 Top field paired with field seq flag shall be equal to 1 previous bottom field in output order
- 57 -Bottom field paired with field seq flag shall be equal to 1 previous top field in output order 11 Top field paired with next field seq flag shall be equal to 1 bottom field in output order 12 Bottom field paired with next field seq flag shall be equal to 1 top field in output order Semantics of the pic struct syntax element pic struct indicates whether a picture should be displayed as a frame or as one or more fields and, for the display of frames when fixed pic rate within cvs flag is equal to 1, may indicate a frame 5 doubling or tripling repetition period for displays that use a fixed frame refresh interval equal to DpbOutputElementalInterval[ n] as given by Equation E-73. The interpretation of pic struct is specified in Table D.2. Values of pic struct that are not listed in Table D.2 are reserved for future use by ITU-T I ISO/IEC and shall not be present in bitstreams conforming to this version of this Specification. Decoders shall ignore reserved values of pic struct.
10 When present, it is a requirement of bitstream conformance that the value of pic struct shall be constrained such that exactly one of the following conditions is true:
¨ The value of pic struct is equal to 0, 7 or 8 for all pictures in the CVS.
¨ The value of pic struct is equal to 1,2, 9, 10, 11 or 12 for all pictures in the CVS.
¨ The value of pic struct is equal to 3, 4, 5 or 6 for all pictures in the CVS.
When fixed pic rate within cvs flag is equal to 1, frame doubling is indicated by pic struct equal to 7, which indicates that the frame should be displayed two times consecutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[ n] as given by Equation E-73, and frame tripling is indicated by pic struct equal to 8, which indicates that the frame should be displayed three times consecutively on displays with a frame refresh interval equal to DpbOutputElementalInterval[ n] as given by Equation E-73.
- 58 -NOTE 3 ¨ Frame doubling can be used to facilitate the display, for example, of 25 Hz progressive-scan video on a 50 Hz progressive-scan display or 30 Hz progressive-scan video on a 60 Hz progressive-scan display. Using frame doubling and frame tripling in alternating combination on every other frame can be used to facilitate the display of 24 Hz progressive-scan video on a 60 Hz progressive-scan display.
The nominal vertical and horizontal sampling locations of samples in top and bottom fields for 4:2:0, 4:2:2 and 4:4:4 chroma formats are shown in Figure D.1, Figure D.2, and Figure D.3, respectively.
Association indicators for fields (pic struct equal to 9 through 12) provide hints to associate fields of complementary parity together as frames. The parity of a field can be top or bottom, and the parity of two fields is considered complementary when the parity of one field is top and the parity of the other field is bottom.
When frame field info present flag is equal to 1, it is a requirement of bitstream conformance that the constraints specified in the third column of Table D.2 shall apply.
NOTE 4 ¨ When frame field info present flag is equal to 0, then in many cases default values may be inferred or indicated by other means. In the absence of other indications of the intended display type of a picture, the decoder should infer the value of pic struct as equal to 0 when frame field info present flag is equal to 0.
What is claimed is:
- 59 -

Claims (6)

PCT/US2021/037449
1. A method for processing an encoded video stream using a processor, the method comprising:
receiving a coded bitstream comprising an encoded picture section including an encoding of a sequence of video pictures and a signaling section including shutter interval parameters, wherein the shutter interval parameters comprise:
a shutter interval time-scale parameter indicating the number of time units passing in one second;
a fixed-shutter-interval-duration flag indicating whether shutter interval duration information is fixed for all pictures in the encoded picture section; and if the fixed-shutter-interval-duration flag indicates that the shutter interval duration information is fixed, then the signaling section includes a shutter interval clock-ticks parameter indicating a number of time units of a clock operating at the frequency of the shutter interval time-scale parameter, wherein the shutter interval clock-ticks parameter divided by the shutter interval time-scale parameter indicates an exposure duration value for all the video pictures in the encoded picture section, else, the shutter interval parameters includes an array of one or more sub-layer shutter interval clock-ticks parameters indicating a number of time units of a clock at the frequency of the shutter interval time-scale parameter for one or more sub-layers in the encoded picture section, wherein, for a first sub-layer in the encoded picture section, a corresponding sub-layer shutter interval clock-ticks parameter divided by the shutter interval time-scale parameter indicates the exposure duration value for all the video pictures in the first sub-layer of the encoded picture section; and decoding the sequence of video pictures based on the shutter interval parameters.
2. The method of claim 1, wherein the shutter interval parameters further include a shutter-interval sub-layer index parameter specifying a sub-layer index of a current picture.
3. The method of claim 2, wherein the encoded picture section comprises two or more access units, and the shutter-interval sub-layer index parameter is 0 for a first access unit among the two or more access units and non-zero otherwise.
4. The method of claim 2, when the shutter-interval sub-layer index parameter is 0 when the fixed-shutter-interval-duration flag indicates that the shutter interval duration information is fixed.
5. The method of claim 1, wherein if the fixed-shutter-interval-duration flag indicates that the shutter interval duration information is not fixed, then the signaling section includes a parameter indicating a number of total elements in the array of one or more sub-layer shutter interval clock-ticks parameters.
6. The method of claim 1, wherein the signaling section comprises a supplemental enhancement information (SEI) messaging section or a video user information (VUI) messaging section.
CA3187271A 2019-03-11 2021-06-15 Frame-rate scalable video coding Active CA3187271C (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CA3192296A CA3192296A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3231109A CA3231109A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3231105A CA3231105A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding

Applications Claiming Priority (9)

Application Number Priority Date Filing Date Title
US201962816521P 2019-03-11 2019-03-11
US201962850985P 2019-05-21 2019-05-21
US201962883195P 2019-08-06 2019-08-06
US201962904744P 2019-09-24 2019-09-24
US16/901,911 US10999585B2 (en) 2019-03-11 2020-06-15 Frame-rate scalable video coding
US16/901,911 2020-06-15
US17/212,701 US11323728B2 (en) 2019-03-11 2021-03-25 Frame-rate scalable video coding
US17/212,701 2021-03-25
PCT/US2021/037449 WO2021257578A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding

Related Child Applications (3)

Application Number Title Priority Date Filing Date
CA3231109A Division CA3231109A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3192296A Division CA3192296A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3231105A Division CA3231105A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding

Publications (2)

Publication Number Publication Date
CA3187271A1 true CA3187271A1 (en) 2021-12-23
CA3187271C CA3187271C (en) 2024-05-28

Family

ID=70166175

Family Applications (4)

Application Number Title Priority Date Filing Date
CA3231105A Pending CA3231105A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3187271A Active CA3187271C (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3231109A Pending CA3231109A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3192296A Pending CA3192296A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CA3231105A Pending CA3231105A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding

Family Applications After (2)

Application Number Title Priority Date Filing Date
CA3231109A Pending CA3231109A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding
CA3192296A Pending CA3192296A1 (en) 2019-03-11 2021-06-15 Frame-rate scalable video coding

Country Status (18)

Country Link
US (12) US11523127B2 (en)
EP (6) EP3850847B1 (en)
JP (6) JP7087100B2 (en)
KR (5) KR102311046B1 (en)
CN (14) CN114666594A (en)
BR (2) BR112020019362A2 (en)
CA (4) CA3231105A1 (en)
DK (2) DK4064706T3 (en)
ES (2) ES2922799T3 (en)
FI (1) FI4064706T3 (en)
HR (2) HRP20220863T1 (en)
HU (2) HUE059347T2 (en)
IL (4) IL289965B (en)
MX (4) MX2023011520A (en)
PH (1) PH12020551511A1 (en)
PL (2) PL3850847T3 (en)
PT (2) PT4064706T (en)
WO (2) WO2020185853A2 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110198474B (en) * 2018-02-27 2022-03-15 中兴通讯股份有限公司 Code stream processing method and device
CN114666594A (en) 2019-03-11 2022-06-24 杜比实验室特许公司 Frame rate scalable video coding
CN114208202A (en) 2019-08-06 2022-03-18 杜比实验室特许公司 Video coding with scalable canvas size
US11812165B2 (en) * 2021-06-16 2023-11-07 Mediatek Inc. Method and apparatus for dynamically changing frame rate of sensor output frames according to whether motion blur condition is met

Family Cites Families (49)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4509917B2 (en) 2005-11-21 2010-07-21 株式会社メガチップス Image processing apparatus and camera system
US8698908B2 (en) 2008-02-11 2014-04-15 Nvidia Corporation Efficient method for reducing noise and blur in a composite still image from a rolling shutter camera
JP2011217272A (en) 2010-04-01 2011-10-27 Canon Inc Video processing apparatus, and method of controlling the same
US8736700B2 (en) * 2010-09-30 2014-05-27 Apple Inc. Techniques for synchronizing audio and video data in an image signal processing system
JP2012195734A (en) * 2011-03-16 2012-10-11 Sony Corp Solid state imaging apparatus, imaging apparatus, electronic apparatus, and solid state imaging apparatus driving method
EP2716041A4 (en) 2011-05-31 2014-10-15 Dolby Lab Licensing Corp Video compression implementing resolution tradeoffs and optimization
US10237565B2 (en) 2011-08-01 2019-03-19 Qualcomm Incorporated Coding parameter sets for various dimensions in video coding
US9196016B2 (en) 2012-01-18 2015-11-24 Linkedin Corporation Systems and methods for improving video stutter in high resolution progressive video
JP6226863B2 (en) 2012-04-13 2017-11-08 三菱電機株式会社 Image encoding device, image decoding device, image encoding method, and image decoding method
US9912941B2 (en) 2012-07-02 2018-03-06 Sony Corporation Video coding system with temporal layers and method of operation thereof
US10110890B2 (en) 2012-07-02 2018-10-23 Sony Corporation Video coding system with low delay and method of operation thereof
US20140003534A1 (en) 2012-07-02 2014-01-02 Sony Corporation Video coding system with temporal scalability and method of operation thereof
US9374581B2 (en) 2013-01-07 2016-06-21 Qualcomm Incorporated Signaling of picture order count to timing information relations for video timing in video coding
US10003815B2 (en) 2013-06-03 2018-06-19 Qualcomm Incorporated Hypothetical reference decoder model and conformance for cross-layer random access skipped pictures
BR112015030508B1 (en) 2013-06-12 2023-11-07 Mitsubishi Electric Corporation IMAGE CODING AND IMAGE DECODING DEVICES AND METHODS
JP6605789B2 (en) 2013-06-18 2019-11-13 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ Transmission method, reception method, transmission device, and reception device
JP6571314B2 (en) 2013-06-18 2019-09-04 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America Sending method
JP6625318B2 (en) 2013-08-29 2019-12-25 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America Transmission method and reception method
US10063886B2 (en) 2013-10-07 2018-08-28 Vid Scale, Inc. Combined scalability processing for multi-layer video coding
JP6538324B2 (en) 2013-10-18 2019-07-03 パナソニック株式会社 Image coding method and image coding apparatus
US8830367B1 (en) 2013-10-21 2014-09-09 Gopro, Inc. Frame manipulation to reduce rolling shutter artifacts
JP6565679B2 (en) * 2013-11-22 2019-08-28 ソニー株式会社 Transmitting apparatus, transmitting method, receiving apparatus, and receiving method
JP6652320B2 (en) 2013-12-16 2020-02-19 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America Transmission method, reception method, transmission device, and reception device
US9860540B2 (en) 2014-01-03 2018-01-02 Qualcomm Incorporated Inference of nooutputofpriorpicsflag in video coding
US10368097B2 (en) 2014-01-07 2019-07-30 Nokia Technologies Oy Apparatus, a method and a computer program product for coding and decoding chroma components of texture pictures for sample prediction of depth pictures
US10567765B2 (en) 2014-01-15 2020-02-18 Avigilon Corporation Streaming multiple encodings with virtual stream identifiers
WO2015115946A1 (en) * 2014-01-30 2015-08-06 Telefonaktiebolaget L M Ericsson (Publ) Methods for encoding and decoding three-dimensional video content
CA2942838C (en) 2014-03-17 2019-10-01 Nokia Technologies Oy An apparatus, a method and a computer program for video coding and decoding
US10187659B2 (en) 2014-04-25 2019-01-22 Sharp Kabushiki Kaisha Image decoding device, image coding device, and coding data transform device
KR20170015905A (en) 2014-05-30 2017-02-10 톰슨 라이센싱 High frame rate tiling compression technique
JP2016005244A (en) * 2014-06-19 2016-01-12 株式会社ニコン Image encoding device, image decoding device, image encoding method and image decoding method
JP6510039B2 (en) 2014-10-02 2019-05-08 ドルビー ラボラトリーズ ライセンシング コーポレイション Dual-end metadata for judder visibility control
CN105744251A (en) 2014-12-11 2016-07-06 深圳市华天瑞彩科技开发有限公司 Shutter type LED three-dimensional display control system
CN107431845B (en) 2015-03-31 2021-03-26 松下电器(美国)知识产权公司 Transmission method, reception method, transmission device, and reception device
WO2016185947A1 (en) 2015-05-19 2016-11-24 ソニー株式会社 Image processing device, image processing method, reception device and transmission device
JP6809471B2 (en) 2015-09-10 2021-01-06 ソニー株式会社 Transmitter, transmitter, receiver and receiver
KR102267922B1 (en) 2015-09-23 2021-06-22 노키아 테크놀로지스 오와이 How to code 360 degree panoramic video, device and computer program product
US10148983B2 (en) 2015-09-25 2018-12-04 Qualcomm Incorporated Methods and systems of improved video stream switching and random access
EP3363195B1 (en) * 2015-10-19 2023-08-30 HFI Innovation Inc. Method and apparatus for decoded picture buffer management in video coding system using intra block copy
US9681111B1 (en) 2015-10-22 2017-06-13 Gopro, Inc. Apparatus and methods for embedding metadata into video stream
JP6588801B2 (en) 2015-10-30 2019-10-09 キヤノン株式会社 Image processing apparatus, image processing method, and program
TWI605705B (en) 2015-11-30 2017-11-11 晨星半導體股份有限公司 Stream decoding method and stream decoding circuit
US10965867B2 (en) 2016-03-03 2021-03-30 Sony Corporation Display control device and display control method
TWI670964B (en) 2016-12-13 2019-09-01 日商夏普股份有限公司 Systems and methods for reducing artifacts in temporal scalable layers of video
US20210136421A1 (en) 2016-12-31 2021-05-06 Sharp Kabushiki Kaisha Systems and methods for reducing artifacts in temporal scalable layers of video
CN111149346B (en) * 2017-09-28 2021-07-13 杜比实验室特许公司 Method, apparatus and medium for encoding and decoding high dynamic range video
US10499009B1 (en) * 2018-09-25 2019-12-03 Pixelworks, Inc. Realistic 24 frames per second output from high frame rate content
US11234044B2 (en) 2018-12-28 2022-01-25 Sony Group Corporation Transmission apparatus, transmission method, encoding apparatus, encoding method, reception apparatus, and reception method
CN114666594A (en) 2019-03-11 2022-06-24 杜比实验室特许公司 Frame rate scalable video coding

Also Published As

Publication number Publication date
ES2952678T3 (en) 2023-11-03
PT4064706T (en) 2023-07-19
WO2021257578A9 (en) 2023-02-02
DK3850847T3 (en) 2022-07-11
EP3850847B1 (en) 2022-05-18
US20220038723A1 (en) 2022-02-03
JP2024020591A (en) 2024-02-14
HUE062299T2 (en) 2023-10-28
DK4064706T3 (en) 2023-07-24
IL289965B (en) 2022-09-01
US11582472B2 (en) 2023-02-14
EP4150909A4 (en) 2023-11-08
PH12020551511A1 (en) 2021-11-22
CN114666594A (en) 2022-06-24
US20240015315A1 (en) 2024-01-11
US20220166991A1 (en) 2022-05-26
PL3850847T3 (en) 2022-08-22
US11871015B2 (en) 2024-01-09
KR20230169394A (en) 2023-12-15
CN114666598A (en) 2022-06-24
WO2020185853A3 (en) 2020-10-29
EP4300974A3 (en) 2024-03-20
PL4064706T3 (en) 2023-08-21
US11936888B1 (en) 2024-03-19
CA3187271C (en) 2024-05-28
CN114666599A (en) 2022-06-24
EP4300975A2 (en) 2024-01-03
KR20230158651A (en) 2023-11-20
CN116668696A (en) 2023-08-29
US20200382802A1 (en) 2020-12-03
US10999585B2 (en) 2021-05-04
JP7087100B2 (en) 2022-06-20
CN117478882A (en) 2024-01-30
JP2024029104A (en) 2024-03-05
CA3192296A1 (en) 2021-12-23
CN117478882B (en) 2024-05-28
CN114666597A (en) 2022-06-24
KR102311046B1 (en) 2021-10-12
US20210211688A1 (en) 2021-07-08
CN111971964B (en) 2022-06-03
JP2023529590A (en) 2023-07-11
EP4064706B1 (en) 2023-06-14
US20240179328A1 (en) 2024-05-30
EP4064706A1 (en) 2022-09-28
JP7353520B2 (en) 2023-09-29
WO2021257578A1 (en) 2021-12-23
US20230171417A1 (en) 2023-06-01
JP2022120065A (en) 2022-08-17
MX2022016068A (en) 2023-02-02
ES2922799T3 (en) 2022-09-20
US20240171760A1 (en) 2024-05-23
IL295131A (en) 2022-09-01
KR20210125110A (en) 2021-10-15
CN111971964A (en) 2020-11-20
KR20230003317A (en) 2023-01-05
CN115702564A (en) 2023-02-14
IL277365A (en) 2020-11-30
CN114666595A (en) 2022-06-24
US11979589B2 (en) 2024-05-07
CA3231109A1 (en) 2021-12-23
EP4236325A2 (en) 2023-08-30
FI4064706T3 (en) 2023-08-07
US20240031587A1 (en) 2024-01-25
US12003741B1 (en) 2024-06-04
PT3850847T (en) 2022-07-01
HRP20220863T1 (en) 2022-10-28
WO2020185853A2 (en) 2020-09-17
US11523127B2 (en) 2022-12-06
IL304034A (en) 2023-08-01
IL277365B (en) 2022-03-01
JP2023169303A (en) 2023-11-29
MX2023004340A (en) 2023-10-16
US20240089474A1 (en) 2024-03-14
CN114845111A (en) 2022-08-02
CN114666596A (en) 2022-06-24
CN114845109A (en) 2022-08-02
US12015793B1 (en) 2024-06-18
US20230073388A1 (en) 2023-03-09
HUE059347T2 (en) 2022-11-28
US20240080465A1 (en) 2024-03-07
CA3231105A1 (en) 2021-12-23
CN116668696B (en) 2024-05-10
EP3850847A2 (en) 2021-07-21
MX2023011519A (en) 2023-10-05
CN115702564B (en) 2023-10-31
EP4300975A3 (en) 2024-03-27
US11818372B2 (en) 2023-11-14
IL295131B1 (en) 2023-08-01
EP4300974A2 (en) 2024-01-03
MX2023011520A (en) 2024-03-26
JP7411727B2 (en) 2024-01-11
HRP20230760T1 (en) 2023-11-10
EP4236325A3 (en) 2023-10-11
CN117014611B (en) 2024-03-15
CN114845110A (en) 2022-08-02
BR112022025555A2 (en) 2023-01-03
US11979588B2 (en) 2024-05-07
IL295131B2 (en) 2023-12-01
KR102502747B1 (en) 2023-02-23
JP2022511162A (en) 2022-01-31
US11323728B2 (en) 2022-05-03
EP4150909A1 (en) 2023-03-22
BR112020019362A2 (en) 2021-09-21
KR20210055025A (en) 2021-05-14
CN117014611A (en) 2023-11-07
IL289965A (en) 2022-03-01

Similar Documents

Publication Publication Date Title
CA3187271C (en) Frame-rate scalable video coding
KR20230026549A (en) Frame-rate scalable video coding

Legal Events

Date Code Title Description
EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214

EEER Examination request

Effective date: 20221214