WO2022190446A1 - 制御装置、制御方法、および、プログラム - Google Patents
制御装置、制御方法、および、プログラム Download PDFInfo
- Publication number
- WO2022190446A1 WO2022190446A1 PCT/JP2021/037485 JP2021037485W WO2022190446A1 WO 2022190446 A1 WO2022190446 A1 WO 2022190446A1 JP 2021037485 W JP2021037485 W JP 2021037485W WO 2022190446 A1 WO2022190446 A1 WO 2022190446A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- content
- type
- information
- control
- type information
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000000694 effects Effects 0.000 claims abstract description 132
- 238000012545 processing Methods 0.000 claims description 24
- 238000010801 machine learning Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 19
- 238000012549 training Methods 0.000 description 12
- 238000001914 filtration Methods 0.000 description 10
- 230000002123 temporal effect Effects 0.000 description 10
- 238000011156 evaluation Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000033001 locomotion Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 206010024796 Logorrhoea Diseases 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4318—Generation of visual interfaces for content selection or interaction; Content or additional data rendering by altering the content in the rendering process, e.g. blanking, blurring or masking an image region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
- H04N21/4662—Learning process for intelligent management, e.g. learning user preferences for recommending movies characterized by learning algorithms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/485—End-user interface for client configuration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/485—End-user interface for client configuration
- H04N21/4854—End-user interface for client configuration for modifying image parameters, e.g. image brightness, contrast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
- H04N21/8405—Generation or processing of descriptive data, e.g. content descriptors represented by keywords
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
- H04N5/57—Control of contrast or brightness
Definitions
- the present disclosure relates to control devices, control methods, and programs.
- the present disclosure provides a control device or the like that appropriately controls the presentation effect based on the type of content.
- the control device includes an acquisition unit that acquires content and first type information indicating the type of the content, and performs type determination processing on the content acquired by the acquisition unit, When the first type information and the second type information match with the determination unit that acquires the second type information indicating the type of the content, the first type information and the second type information do not match. and a generating unit configured to generate and output control information for increasing the strength of the presentation effect to be given when presenting the content than when presenting the content.
- a control method acquires content, acquires first type information indicating the type of the content, and performs type determination processing on the obtained content, thereby obtaining the first type information indicating the type of the content.
- the content is presented rather than when the first-type information and the second-type information do not match.
- the control device can appropriately control the presentation effect based on the type of content.
- FIG. 1 is an explanatory diagram showing the appearance of a device provided with a control device according to an embodiment.
- FIG. 2 is a block diagram showing the functional configuration of the control device according to the embodiment.
- FIG. 3 is an explanatory diagram illustrating an example of a type acquired by an acquisition unit and a type determined by a determination unit for content according to the embodiment.
- FIG. 4 is an explanatory diagram illustrating an example of training data used for learning for type determination by the determining unit according to the embodiment.
- FIG. 5 is an explanatory diagram of an example of type information indicating a result of type determination by the determining unit according to the embodiment.
- FIG. 6 is an explanatory diagram illustrating an example of temporal changes in match or mismatch between an acquisition result obtained by an obtaining unit and a type determination result obtained by a determination unit according to the embodiment.
- FIG. 7 is an explanatory diagram illustrating an example of the strength of the presentation effect indicated by the control information generated by the generating unit according to the embodiment.
- FIG. 8 is an explanatory diagram of frames used for calculation of filter processing executed by the generation unit according to the embodiment.
- FIG. 9 is an example of indices used for filtering performed by the generation unit according to the embodiment.
- 10 is an example of the strength of the presentation effect obtained by the filtering process executed by the generating unit according to the embodiment;
- FIG. 11 is an explanatory diagram illustrating an example of an image of an operation bar used for user setting of presentation effects according to the embodiment.
- FIG. 12 is a flow diagram showing a control method of the control device according to the embodiment.
- the inventors of the present application have found that the following problems arise with conventional control of presentation effects based on the type of content.
- the types of content are classified, for example, based on official program information (also called SI (Service Information)) given to broadcast programs.
- SI Service Information
- the type is, for example, sports, music, talk, cinema, or the like.
- part of a broadcast program that mainly includes soccer matches includes scenes where soccer players speak in the studio.
- the broadcast programs as a whole are classified into the sports category, and presentation effects suitable for the sports category are given to the overall broadcast programs.
- presentation effects suitable for the sports category are given to the overall broadcast programs.
- a presentation effect suitable for the content of the talk type it is appropriate to give a presentation effect suitable for the content of the talk type, but a presentation effect suitable for the content of the sports type is given, in other words, an inappropriate presentation. effect is granted.
- the present disclosure provides a control device or the like that appropriately controls the presentation effect based on the type of content.
- a control device includes an acquisition unit that acquires content and first type information indicating a type of the content, and performs type determination processing on the content acquired by the acquisition unit.
- the determination unit that acquires the second type information indicating the type of the content and the first type information and the second type information match, the first type information and the second type information and a generating unit configured to generate and output control information for increasing the strength of the presentation effect to be given when presenting the content than when the content is not matched.
- the control device since the control device uses the type information acquired by the acquisition unit and the type information acquired by the determination unit through the type determination process, the presentation effect corresponding to the type determined more appropriately can be obtained more effectively. It can be controlled to apply at high intensity. Therefore, the control device can appropriately control the presentation effect based on the content type.
- the determination unit inputs the content to a recognition model constructed by machine learning, and determines the type information of the content output by inputting the content to the recognition model, It may be acquired as the second type information.
- the determination unit acquires the type of content using the recognition model constructed by machine learning, so that the type of content can be acquired more appropriately. Therefore, the control device can more appropriately control the presentation effect based on the content type.
- the first type information may indicate the type of the entire content
- the determination unit may determine the type of each of a plurality of partial contents included in the content.
- the control device increases the strength of the presentation effect to be imparted when presenting a partial content whose type information of the entire content matches the type of the partial content among the plurality of partial content included in the content. to control. Therefore, the control device can appropriately control the presentation effect based on the content type for each partial content.
- the acquisition unit may acquire information set as information indicating the type of content from a device different from the control device as the first type information.
- the control device acquires the information set as the information indicating the content type as the first type information, so that the first type information can be obtained more easily.
- the control device can obtain the first type information without performing the process of determining the type of content. Therefore, power consumption required for the processing or resources such as hardware used for the processing are unnecessary. Therefore, the control device can more easily and appropriately control the presentation effect based on the type of content.
- the acquisition unit may acquire, as the first type information, the content type information obtained by analyzing the acquired content.
- the control device acquires the information obtained by analyzing the content as the first type information, so even if there is no other device that provides the information indicating the content type , the first type information can be obtained. Therefore, the control device can appropriately control the presentation effect based on the content type.
- control information may include information indicating the strength of the presentation effect when presenting the content in chronological order.
- control device can control the presentation effect in chronological order using the control information shown in chronological order. Therefore, the control device can more appropriately control the presentation effect based on the content type.
- the generating unit when generating the control information, may perform a process of suppressing a sudden change in the strength of the presentation effect when presenting the content.
- the control device controls the presentation effect using the control information in which a rapid change in the intensity of the presentation effect is suppressed. be. Therefore, the control device can appropriately control the presentation effect based on the type of content while suppressing its rapid change.
- the generation unit has correspondence information in which type information indicating a type of content and presentation effects to be imparted when presenting the content of the type are associated in advance, and the control information is generated by: At the time of generation, control information for imparting a presentation effect previously associated with the first type information may be generated as the control information.
- control device can apply the presentation effect pre-associated with the type of content while controlling the intensity thereof. Therefore, the control device can appropriately control the presentation effect while appropriately imparting the presentation effect corresponding to the type of content.
- the generation unit may generate, as the control information, control information for increasing the strength of at least one of a sound effect and a video effect as a presentation effect when presenting the content.
- control device controls at least one of the sound effect and the video effect as the presentation effect. Therefore, the control device can appropriately control the sound effect or the video effect based on the content type.
- the generation unit may receive an operation from the user to set the intensity range of the presentation effect, and generate the control information for controlling the presentation effect within the intensity range set by the operation.
- the control device controls the strength of the presentation effect within the range received from the user. Therefore, the control device can more appropriately control the presentation effect based on the type of content, with the intensity reflecting the user's preference for the strength of the presentation effect.
- a control method acquires content, acquires first type information indicating a type of the content, and performs a type determination process on the acquired content to determine whether the content Second type information indicating a type is acquired, and when the first type information and the second type information match, the content is more likely than the case where the first type information and the second type information do not match.
- control method has the same effect as the above control device.
- a program according to one aspect of the present disclosure is a program that causes a computer to execute the control method described above.
- control method has the same effect as the above control device.
- FIG. 1 is an explanatory diagram showing the appearance of a device provided with a control device 10 according to this embodiment.
- An example of a device that includes the control device 10 is the television receiver 1 .
- the television receiver 1 receives a signal containing content including sound and video, and presents the sound and video included in the content.
- the signal includes, for example, a signal contained in a broadcast wave transmitted from a broadcasting station, a signal transmitted from various transmission sources via a communication line, or a signal transmitted by an external device.
- Various transmission sources include, for example, a server for a video providing service on the Internet.
- the external device is, for example, a recording device, a computer, a game machine, or the like.
- a case where the television receiver 1 receives a signal included in a broadcast wave will be described as an example.
- the television receiver 1 includes a tuner (not shown), a speaker 5, and a screen 6.
- the speaker 5 outputs sound obtained through the tuner from a signal included in the broadcast wave, and the sound is included in the broadcast wave. An image obtained from the signal through the tuner is displayed on the screen 6 .
- Content includes data or signals of a certain length of time, including at least video. Content may also be data of some length of time, including sound and video.
- the content may correspond to one broadcast program, or may correspond to a portion of a predetermined length of time included in one broadcast program.
- the time length of the content is, for example, equal to or longer than the time corresponding to one frame of video and is equal to or shorter than several seconds to several hours.
- the content may further include meta information.
- the meta information may include official program information (also called SI (Service Information)).
- control device 10 receives broadcast waves, stores content, and then plays back the content. may be provided on the machine.
- the control device 10 outputs control information for controlling presentation effects to be given when the television receiver 1 reproduces content.
- the control device 10 acquires the broadcast wave received by the television receiver 1, and determines which type of content among a plurality of predetermined types the content obtained from the signal included in the broadcast wave is. Then, the control device 10 generates and outputs control information for controlling presentation effects when presenting the content.
- the control information includes information indicating the type of content and information indicating the strength of the presentation effect when presenting the content (described later).
- Types of content include, for example, sports, music, talk, cinema, and the like. Note that the type that does not correspond to any of the above types will be referred to as default.
- FIG. 2 is a block diagram showing the functional configuration of the control device 10 according to this embodiment.
- the control device 10 includes an acquisition unit 11, a determination unit 12, and a generation unit 13.
- the control device 10 is also connected to a sound control section 21 and a video control section 22 provided in the television receiver 1 .
- the functional units provided in the control device 10 can be realized by a processor (eg, a CPU (Central Processing Unit)) (not shown) provided in the control device 10 executing a predetermined program using a memory (not shown).
- a processor eg, a CPU (Central Processing Unit)
- the acquisition unit 11 is a functional unit that acquires content and also acquires type information (corresponding to first type information) indicating the type of the content.
- the content acquired by the acquisition unit 11 is content acquired by the television receiver 1 from broadcast waves or the like. Acquisition unit 11 provides the acquired content to determination unit 12 and generation unit 13 .
- the type information acquired by the acquisition unit 11 is information indicating the type of the entire content acquired by the acquisition unit 11, in other words, information given to the entire content.
- the type information acquired by the acquisition unit 11 may be meta information (for example, SI) set by a device external to the control device 10 as information indicating the type of content acquired by the acquisition unit 11 .
- the external device may be a device owned by a television station that provides broadcast programs, or a device owned by a third party that generates meta information. may be a device dedicated to
- the type information is acquired from the HDMI (registered trademark) content type (Content Type).
- the type information acquired by the acquisition unit 11 may be type information obtained by analyzing the content acquired by the acquisition unit 11 .
- the acquisition unit 11 executes a process of analyzing features in the video data, audio data and meta information of the content. Specifically, the acquisition unit 11 detects a human line of sight included in the video of the content, detects the movement of an object included in the video of the content, detects a specific sound included in the sound of the content, or , the type of the content is specified by executing a pattern detection process of an object included in the image of the content.
- Well-known image recognition technology and sound recognition technology can be used for analysis of video data and sound data.
- the acquisition unit 11 determines the type of content based on detection of predetermined information or data in the video, sound, or meta information of the content.
- the type of content is determined to be "talk”.
- the content type is determined to be "sports”
- the content type is determined to be "talk”.
- the sound detection process when a singing voice or a sound played by a musical instrument is detected, the content type is determined as "music”.
- the object pattern detection process when an image of a uniform is detected, the type of content is determined to be "sports”, and when an image of a musical instrument is detected, the type of content is determined to be "music". judge.
- the determination unit 12 is a functional unit that acquires type information (equivalent to second type information) indicating the type of content by performing type determination processing on the content acquired by the acquisition unit 11 .
- the determination unit 12 inputs the content to a recognition model constructed in advance by machine learning, and acquires the type information of the content output by inputting the content as the second type information. good.
- the determination unit 12 determines the type of each of a plurality of partial contents included in the content.
- the partial content is a portion having a predetermined number of frames included in the content, for example, a portion having 1 frame, 10 frames, or 30 frames.
- the partial content may be a portion that has a predetermined time length (for example, 1 second, 5 seconds, or 10 seconds) included in the content.
- the determining unit 12 acquires the type of content acquired by the acquiring unit 11 by a method different from the method by which the acquiring unit 11 acquires the type of content through the type determination process. More specifically, it can be said that the determination unit 12 acquires the type of content acquired by the acquisition unit 11 without using meta information, for example.
- the generating unit 13 is a functional unit that generates and outputs control information for controlling the strength of the presentation effect when presenting content.
- the generation unit 13 performs more To generate control information for increasing the intensity of a presentation effect to be given when content is presented.
- Presentation effects include at least one of sound effects and visual effects.
- the control information is output to the sound control section 21 and the video control section 22 .
- control information includes information indicating the strength of the presentation effect to be given when presenting the content in chronological order.
- the control information imparts a stronger presentation effect when presenting the partial content when the first type information about the content matches the second type information for each of the plurality of partial content included in the content. It is shown that.
- the generation unit 13 has correspondence information in which type information indicating the type of content and presentation effects to be given when presenting content of the type are associated in advance. Then, when generating the control information, the generation unit 13 generates the control information for imparting the presentation effect preliminarily associated with the first type information.
- the association information has, for example, the following information as associations between type information and sound effects.
- a sound effect that increases the spread of sound and changes the output direction of the sound so that the viewer feels as if they are surrounded by the sound is associated.
- audio signal processing is applied to change the way people hear sounds so as to increase the spread of sound, and vocals are emphasized.
- a sound effect that changes the audio amplitude is associated with each output frequency band.
- the content of the type “talk” is associated with a sound effect that changes the audio amplitude for each output frequency band so that the viewer can easily hear the performer's voice.
- association information has, for example, the following information as associations between type information and video effects.
- “sports” type content is associated with a video effect that increases the brightness and sharpness of the video so as to make the video brighter and more vivid.
- content of the type “cinema” is associated with a video effect that increases the contrast while suppressing the brightness of the video so that the texture can be expressed richly.
- the generation unit 13 may perform processing for suppressing rapid changes in the strength of the presentation effect when generating the control information.
- the above processing is also called filtering processing.
- the above processing is so-called low-pass filter processing, and can also be called noise removal processing or smoothing processing.
- the sound control unit 21 is a functional unit that acquires control information output by the generation unit 13 and controls sound output from the speaker 5 based on the control information.
- the sound control unit 21 controls output of the sound included in the content acquired by the acquisition unit 11 from the speaker 5 .
- the sound control unit 21 processes the sound to be output so as to apply a sound effect according to the intensity of the presentation effect included in the control information.
- the video control unit 22 is a functional unit that acquires control information output by the generation unit 13 and controls display of images on the screen 6 based on the control information.
- the image control unit 22 controls displaying on the screen 6 an image included in the content acquired by the acquisition unit 11 . At that time, the image control unit 22 processes the image to be displayed so as to apply an image effect according to the intensity included in the control information.
- control device 10 In the following, the processing of the control device 10 will be described in more detail.
- FIG. 3 is an explanatory diagram showing an example of the type acquired by the acquisition unit 11 and the type determined by the determination unit 12 for content according to the present embodiment.
- the content shown in FIG. 3 is an example of the content acquired by the acquisition unit 11, and is the content of a soccer match broadcast program. More specifically, the contents include partial contents of opening, competition, CM (commercial message), competition, audience, competition, and interview in this order.
- CM commercial message
- the acquisition unit 11 acquires information indicating "sports” as the type of the entire content (hereinafter also simply referred to as "sports").
- the determination unit 12 acquires information indicating the type of each of the plurality of partial contents by determining the type of each of the plurality of partial contents included in the content. Specifically, the determination unit 12 acquires "default” as the type of the partial content of the opening or CM, "sports” as the type of the partial content of the competition or the audience, and "default” as the type of the partial content of the interview. Get Talk.
- FIG. 4 is an explanatory diagram showing an example of training data used for learning for type determination by the determining unit 12 according to the present embodiment.
- the training data shown in FIG. 4 is training data in which one partial content and one type are associated.
- partial content including an image showing a player playing soccer is associated with "sports" as the type of the partial content.
- partial content including an image showing a singer singing on stage is associated with "music" as the type of the partial content.
- partial content including an image showing a performer having a dialogue is associated with "talk" as the type of the partial content.
- training data may include not only images but also sounds.
- the training data may include thousands to tens of thousands of contents.
- the type associated with each partial content is one of a plurality of predetermined types.
- the plurality of predetermined types include, but are not limited to, "sports,” “music,” and "talk.”
- the determination unit 12 builds a recognition model in advance by machine learning using training data.
- the recognition model is, for example, a neural network recognition model.
- the determination unit 12 extracts features of the image or sound of the input partial content by training using the training data, and outputs the type corresponding to the input partial content.
- a recognition model is constructed by adjusting the coefficients of the nodes.
- the recognition model trained in this way outputs type information indicating the type of the content based on the image and sound features of the input partial content.
- the type information output by the recognition model is, for example, information specifying which of a plurality of predetermined types the input partial content belongs to, and this case will be described as an example.
- the type information to be output may be information including a score that is the probability that the input partial content is classified into each of a plurality of predetermined types.
- FIG. 5 is an explanatory diagram showing an example of type information indicating the result of type determination by the determination unit 12 according to this embodiment.
- the determination unit 12 acquires the type information output by inputting the partial content included in the content acquired by the acquisition unit 11 into the recognition model.
- the recognition model when the partial content 31 shown in FIG. 5 is input to the recognition model, the recognition model outputs "sports" as the type of the input partial content 31 .
- FIG. 6 is an explanatory diagram showing an example of temporal changes in match or mismatch between the acquisition result by the acquisition unit 11 and the type determination result by the determination unit 12 according to the present embodiment. Specifically, FIG. 6 is a graph showing in chronological order whether or not the type of partial content determined by the determination unit 12 matches the type of the entire content acquired by the acquisition unit 11 .
- the type of the entire content is "sports”
- the type of the partial content is determined to be “sports” by the determining unit 12
- the type is “match” in the period corresponding to the partial content
- the type of the partial content is “match”.
- the type is “mismatched” in the period corresponding to the partial content.
- the scale of the horizontal axis in FIG. 6 is arbitrary, for example, one scale corresponds to the time of 20 frames.
- FIG. 7 is an explanatory diagram showing an example of the intensity I of the presentation effect indicated in the control information by the generation unit 13 according to the present embodiment.
- the generation unit 13 generates control information indicating the intensity I of the presentation effect based on the match or mismatch of the types shown in FIG.
- the intensity I is set to 100% during the period when the types match, and the intensity I during the period when the types do not match is set to 0%.
- a strength of 0% means that no special presentation effect is given, in other words, normal presentation is made. Note that 100% and 0% in the above are examples, and in the intensity information, the intensity I in the period in which the types match is set higher than the intensity I in the period in which the types do not match.
- the intensity may be changed by combining the first type information and the second type information.
- the control information shown in FIG. 7 controls the strength of the acoustic effect by the sound control unit 21, and the speaker 5 outputs the sound to which the controlled strength of the acoustic effect is added. Further, the intensity of the image effect is controlled by the image control unit 22 according to the output control information, and the image to which the image effect of the controlled intensity is applied is displayed on the screen 6 .
- control device 10 can appropriately control the presentation effect based on the type of content.
- a method using a weighted moving average can be used for filtering.
- FIG. 8 is an explanatory diagram showing frames used for calculation of filter processing executed by the generation unit 13 according to the present embodiment.
- FIG. 9 is an example of indexes used for filtering executed by the generation unit 13 according to the present embodiment.
- FIG. 10 is an example of the strength of the presentation effect obtained by the filtering process executed by the generation unit 13 according to the present embodiment.
- a frame at time t shown in FIG. 8 is a frame that is a partial content subject to type determination.
- Weighted moving average using k+1 frames from time t ⁇ k to time t in the filtering process is multiplied by Gain, which is a number greater than 0 is used as the evaluation value E.
- Gain is a number greater than 0 and indicates a calculation interval.
- Gain functions as a parameter for adjusting the sensitivity of changes in the strength of the presentation effect. If the evaluation value E exceeds 1, it is set to 1, so that the evaluation value E falls within a range of greater than 0 and 1 or less.
- FIG. 9 shows temporal changes in the evaluation value E calculated in this way.
- the temporal change in the evaluation value E as shown in FIG. 9 corresponds to the temporal change in the intensity I shown in FIG. 7 in which abrupt changes are suppressed.
- the intensity I(t) at time t is expressed as follows using the intensity I(t-1) immediately before time t, that is, at time t-1.
- p is a numerical value greater than 0 and less than 1, and whether the evaluation value E or the intensity I(t ⁇ 1) at time t ⁇ 1 is more heavily reflected in the intensity I(t) at time t functions as a parameter for adjusting the
- Fig. 10 shows temporal changes in intensity I calculated in this way.
- the temporal change in the intensity I shown in FIG. 10 corresponds to the rapid change in the temporal change in the evaluation value E shown in FIG. 9 that is further suppressed.
- the strength of the presentation effect can also be controlled by reflecting the settings made by the user.
- FIG. 11 is an explanatory diagram showing an image 40 as an example of an operation bar used for user setting of presentation effects according to the present embodiment.
- an operation bar image 40 shows an operation bar extending left and right.
- the image 40 has a scale 41 indicating 0, a scale 42 indicating 10, and a marking 43 movable between the scales.
- the mark 43 When the image 40 is displayed on the touch panel display, the mark 43 is moved left and right by the user's touch operation, and indicates a numerical value within the range of 0 to 10 depending on the position of the mark 43.
- the position of the solid line marking 43 indicates the numerical value of seven
- the position of the dashed marking 43 indicates the numerical value of four.
- the generation unit 13 By reading the position of the mark 43 on the operation bar, the generation unit 13 receives the setting of the intensity range of the presentation effect from the user. Then, the generation unit 13 generates control information for controlling the presentation effect within the intensity range set by the above operation.
- the generation unit 13 uses the numerical value read as the position of the mark 43 on the operation bar as the upper limit of the presentation effect. For example, when the mark 43 indicates a numerical value of 7 in the operation bar indicating the range from 0 to 10, the intensity of the presentation effect calculated by the generation unit 13 is multiplied by 0.7 to give a presentation effect, make a presentation.
- control device 10 can provide the presentation effect with an intensity that reflects the user's preference for the intensity of the presentation effect.
- the operation bar is not limited to the example in which it is arranged so as to extend left and right, and may be arranged so as to extend in a vertical or oblique direction. Further, the shape of the operation bar is not limited to the above example, and any shape may be used as long as the user can recognize that the image is used for changing the intensity of the presentation effect. good too.
- buttons or keys when the image 40 is displayed on a normal display that is not a touch panel display, the user can operate buttons or keys to perform operations similar to those described above.
- control device 10 configured as above will be described.
- FIG. 12 is a flow chart showing the control method of the control device 10 according to the embodiment. The control method shown in FIG. 12 may be executed for each frame of content.
- step S101 the acquisition unit 11 acquires content.
- step S102 the acquisition unit 11 acquires type information indicating the type of the entire content of the content acquired in step S101.
- step S103 the determination unit 12 performs type determination processing on the content acquired in step S101, thereby acquiring type information for each of a plurality of partial contents included in the content.
- steps S104, S105 and S111 are executed for each of the plurality of partial contents.
- step S104 the generation unit 13 determines whether or not the type information of the entire content acquired in step S101 matches the type information of each of the plurality of partial contents acquired in step S102. If the above two pieces of type information match (Yes in step S104), the process proceeds to step S105, otherwise (No in step S104), the process proceeds to step S111.
- step S105 the generation unit 13 generates control information for increasing the strength of the presentation effect for the partial content to be processed.
- the control information for increasing the strength of the presentation effect is, in other words, control information for increasing the strength of the presentation effect higher than when the strength of the presentation effect is normal (step S111).
- Control information for increasing the strength of the presentation effect is, for example, control information indicating 100% in FIG.
- step S111 the generation unit 13 generates control information that sets the strength of the presentation effect to normal (that is, does not add a special presentation effect) for the partial content to be processed.
- the control information that sets the strength of the presentation effect to normal is, in other words, control information that sets the strength of the presentation effect to be reduced compared to when the strength of the presentation effect is increased (step S105).
- Control information that sets the strength of the presentation effect to normal is, for example, control information indicating 0% in FIG.
- the generation unit 13 acquires the temporal change of the intensity I illustrated in FIG. 7 by executing step S105 or step S111 for each of the plurality of partial contents.
- step S106 the generating unit 13 performs filtering to suppress rapid changes in the strength of the presentation effect. Thereby, the generation unit 13 acquires the temporal change of the intensity I illustrated in FIG. 10 .
- step S106 does not have to be executed. Note that when step S106 is executed, it is necessary to calculate the intensity of the presentation effect for a predetermined period before the partial content to be processed.
- step S107 the generation unit 13 outputs control information.
- the output control information includes type information indicating the type of the presentation effect and information indicating the intensity I of the presentation effect.
- Intensity I is the intensity I acquired in step S105 or step S111, and is the intensity I after the filtering process is performed when the filtering process in step S106 is performed.
- the control information output in step S107 controls the strength of the sound effect by the sound control unit 21, and the speaker 5 outputs sound accompanied by the sound effect of the controlled strength. Also, the intensity of the image effect by the image control unit 22 is controlled by the output control information, and the image accompanied by the image effect of the controlled intensity is displayed on the screen 6 .
- the present disclosure is applicable to television receivers, recording devices, and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
本実施の形態において、コンテンツの種別に基づく提示効果の制御を適切に行う制御装置について説明する。
5 スピーカ
6 画面
10 制御装置
11 取得部
12 判定部
13 生成部
21 音制御部
22 映像制御部
31 部分コンテンツ
40 画像
41、42 目盛り
43 印
Claims (12)
- コンテンツを取得し、かつ、前記コンテンツの種別を示す第一種別情報を取得する取得部と、
前記取得部が取得した前記コンテンツに対して種別判定処理を行うことで、前記コンテンツの種別を示す第二種別情報を取得する判定部と、
前記第一種別情報と前記第二種別情報とが一致する場合に、前記第一種別情報と前記第二種別情報とが一致しない場合よりも、前記コンテンツを提示する際に付与する提示効果の強度を高くする制御情報を生成して出力する生成部とを備える
制御装置。 - 前記判定部は、前記種別判定処理において、
機械学習によって構築された認識モデルに、前記コンテンツを入力し、
前記認識モデルに前記コンテンツを入力することで出力された前記コンテンツの種別情報を、前記第二種別情報として取得する
請求項1に記載の制御装置。 - 前記第一種別情報は、前記コンテンツ全体の種別を示しており、
前記判定部は、前記コンテンツに含まれる複数の部分コンテンツそれぞれの種別を判定する
請求項2に記載の制御装置。 - 前記取得部は、前記コンテンツの種別を示す情報として設定された情報を前記制御装置と異なる装置から、前記第一種別情報として取得する
請求項1~3のいずれか1項に記載の制御装置。 - 前記取得部は、取得した前記コンテンツを分析することで得られる前記コンテンツの種別情報を、前記第一種別情報として取得する
請求項1~3のいずれか1項に記載の制御装置。 - 前記制御情報は、前記コンテンツを提示する際の提示効果の強度を時系列で示す情報を含む
請求項1~5のいずれか1項に記載の制御装置。 - 前記生成部は、前記制御情報を生成するときに、前記コンテンツを提示する際の提示効果の強度の急激な変化を抑制する処理を施す
請求項1~6のいずれか1項に記載の制御装置。 - 前記生成部は、
コンテンツの種別を示す種別情報と、当該種別のコンテンツを提示する際に付与すべき提示効果とが予め対応付けられた対応付け情報を有しており、
前記制御情報を生成する際には、前記第一種別情報に予め対応付けられた提示効果を付与する制御情報を、前記制御情報として生成する
請求項1~7のいずれか1項に記載の制御装置。 - 前記生成部は、前記コンテンツを提示する際の提示効果として、音響効果および映像効果の少なくとも一方の強度を高くする制御情報を、前記制御情報として生成する
請求項1~8のいずれか1項に記載の制御装置。 - 前記生成部は、提示効果の強度の範囲を設定する操作をユーザから受け、
前記操作により設定される強度の範囲内で提示効果を制御する前記制御情報を生成する
請求項1~9のいずれか1項に記載の制御装置。 - コンテンツを取得し、かつ、前記コンテンツの種別を示す第一種別情報を取得し、
取得した前記コンテンツに対して種別判定処理を行うことで、前記コンテンツの種別を示す第二種別情報を取得し、
前記第一種別情報と前記第二種別情報とが一致する場合に、前記第一種別情報と前記第二種別情報とが一致しない場合よりも、前記コンテンツを提示する際に付与する提示効果の強度を高くする制御情報を生成して出力する
制御方法。 - 請求項11に記載の制御方法をコンピュータに実行させるプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2023505092A JP7499459B2 (ja) | 2021-03-11 | 2021-10-08 | 制御装置、制御方法、および、プログラム |
US18/280,402 US20240155192A1 (en) | 2021-03-11 | 2021-10-08 | Control device, control method, and recording medium |
EP21930293.2A EP4307693A4 (en) | 2021-03-11 | 2021-10-08 | CONTROL DEVICE, CONTROL METHOD AND PROGRAM |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021039093 | 2021-03-11 | ||
JP2021-039093 | 2021-03-11 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022190446A1 true WO2022190446A1 (ja) | 2022-09-15 |
Family
ID=83227697
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/037485 WO2022190446A1 (ja) | 2021-03-11 | 2021-10-08 | 制御装置、制御方法、および、プログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US20240155192A1 (ja) |
EP (1) | EP4307693A4 (ja) |
JP (1) | JP7499459B2 (ja) |
WO (1) | WO2022190446A1 (ja) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002158941A (ja) * | 2000-11-16 | 2002-05-31 | Sharp Corp | 画質制御装置 |
JP2006277232A (ja) | 2005-03-29 | 2006-10-12 | Kddi Corp | 動画像データの分類装置 |
JP2008153895A (ja) * | 2006-12-15 | 2008-07-03 | Canon Inc | 放送受信装置及びその制御方法 |
WO2011037147A1 (ja) * | 2009-09-25 | 2011-03-31 | シャープ株式会社 | 表示装置、プログラム及びプログラムが記録されたコンピュータ読み取り可能な記憶媒体 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120206493A1 (en) * | 2009-10-27 | 2012-08-16 | Sharp Kabushiki Kaisha | Display device, control method for said display device, program, and computer-readable recording medium having program stored thereon |
KR102229156B1 (ko) * | 2014-03-05 | 2021-03-18 | 삼성전자주식회사 | 디스플레이 장치 및 디스플레이 장치의 제어 방법 |
-
2021
- 2021-10-08 EP EP21930293.2A patent/EP4307693A4/en active Pending
- 2021-10-08 JP JP2023505092A patent/JP7499459B2/ja active Active
- 2021-10-08 WO PCT/JP2021/037485 patent/WO2022190446A1/ja active Application Filing
- 2021-10-08 US US18/280,402 patent/US20240155192A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002158941A (ja) * | 2000-11-16 | 2002-05-31 | Sharp Corp | 画質制御装置 |
JP2006277232A (ja) | 2005-03-29 | 2006-10-12 | Kddi Corp | 動画像データの分類装置 |
JP2008153895A (ja) * | 2006-12-15 | 2008-07-03 | Canon Inc | 放送受信装置及びその制御方法 |
WO2011037147A1 (ja) * | 2009-09-25 | 2011-03-31 | シャープ株式会社 | 表示装置、プログラム及びプログラムが記録されたコンピュータ読み取り可能な記憶媒体 |
Non-Patent Citations (1)
Title |
---|
See also references of EP4307693A4 |
Also Published As
Publication number | Publication date |
---|---|
EP4307693A1 (en) | 2024-01-17 |
JP7499459B2 (ja) | 2024-06-14 |
EP4307693A4 (en) | 2024-07-10 |
US20240155192A1 (en) | 2024-05-09 |
JPWO2022190446A1 (ja) | 2022-09-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6573870B2 (ja) | オーディオ分類および処理のための装置および方法 | |
JP2019216408A (ja) | 情報を出力するための方法、及び装置 | |
JP2020003537A5 (ja) | オーディオ抽出装置、オーディオ再生装置、オーディオ抽出方法、オーディオ再生方法、機械学習方法及びプログラム | |
US11011187B2 (en) | Apparatus for generating relations between feature amounts of audio and scene types and method therefor | |
CN109348274B (zh) | 一种直播互动方法、装置及存储介质 | |
JP2016523006A (ja) | ボリューム平準化器コントローラおよび制御方法 | |
JP2016519493A (ja) | 等化器コントローラおよび制御方法 | |
JP6203003B2 (ja) | 信号処理装置、信号処理方法およびプログラム | |
JP7453712B2 (ja) | オーディオ再生方法、装置、コンピュータ可読記憶媒体及び電子機器 | |
US11756571B2 (en) | Apparatus that identifies a scene type and method for identifying a scene type | |
JP2014089420A (ja) | 信号処理装置、方法およびプログラム | |
US7203558B2 (en) | Method for computing sense data and device for computing sense data | |
US20140181273A1 (en) | Information system, information reproducing apparatus, information generating method, and storage medium | |
CN114615534A (zh) | 显示设备及音频处理方法 | |
CN114466242A (zh) | 显示设备及音频处理方法 | |
WO2022190446A1 (ja) | 制御装置、制御方法、および、プログラム | |
CN116996702A (zh) | 演唱会直播处理方法、装置、存储介质和电子设备 | |
JP2006254187A (ja) | 音場判定方法及び音場判定装置 | |
JP7466087B2 (ja) | 推定装置、推定方法、及び、推定システム | |
CN110097618A (zh) | 一种音乐动画的控制方法、装置、车辆及存储介质 | |
US20230353800A1 (en) | Cheering support method, cheering support apparatus, and program | |
JP6619072B2 (ja) | 音合成装置、音合成方法、及びそのプログラム | |
KR20220036210A (ko) | 영상의 음질을 향상시키는 디바이스 및 방법 | |
US20240038207A1 (en) | Live distribution device and live distribution method | |
CN112992186B (zh) | 音频处理方法、装置、电子设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21930293 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2023505092 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18280402 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2021930293 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2021930293 Country of ref document: EP Effective date: 20231011 |