WO2019209006A1 - Method for improving resolution of streaming files - Google Patents
Method for improving resolution of streaming files Download PDFInfo
- Publication number
- WO2019209006A1 WO2019209006A1 PCT/KR2019/004891 KR2019004891W WO2019209006A1 WO 2019209006 A1 WO2019209006 A1 WO 2019209006A1 KR 2019004891 W KR2019004891 W KR 2019004891W WO 2019209006 A1 WO2019209006 A1 WO 2019209006A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video data
- data
- neural network
- learning
- resolution
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/222—Secondary servers, e.g. proxy server, cable television Head-end
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
Definitions
- the present invention relates to a method for improving the resolution of a video file based on artificial intelligence. More specifically, the present invention relates to a method for restoring low quality data to high quality data based on an artificial neural network learned by extracting information about a grid generated in a process of changing image data to low quality in a streaming file.
- An object of the present invention is to improve the resolution of a streaming video file in a simple way.
- the present invention can generate video data and neural network files in the form of divided files so that the resolution of the streamed video data can be improved in real time during video streaming.
- a resolution improving method includes processing a video data in a server providing streaming video data, acquiring grid generation pattern information based on the processed video data, and applying the grid generation pattern information to the grid generation pattern information.
- the present invention has the effect of reducing the storage capacity of the video data, thereby enabling the effective use of the storage space.
- the present invention can reduce the capacity of the video data to perform data communication, it is also possible to reduce the data communication amount and to support a data transmission function including streaming at a higher speed.
- FIG. 1 is a diagram illustrating a configuration of a system for performing resolution improvement according to an exemplary embodiment of the present invention.
- FIG. 2 is a diagram illustrating an example of an image quality improvement operation according to an exemplary embodiment of the present invention.
- FIG. 3 is a block diagram illustrating a configuration of a server according to an exemplary embodiment of the present invention.
- FIG. 4 is a diagram illustrating a configuration of a data processing unit according to an exemplary embodiment of the present invention.
- FIG. 5 is a diagram illustrating a configuration of a neural network learning unit according to an embodiment of the present invention.
- FIG. 6 is a diagram illustrating an example of a size change operation performed by a size change unit according to an exemplary embodiment of the present invention.
- FIG. 7 is a view for explaining an example of a deep learning learning operation according to an embodiment of the present invention.
- FIG. 8 is a diagram illustrating a configuration of a user device according to an exemplary embodiment of the present invention.
- FIG. 9 is a diagram illustrating a generation and transmission process of an image file for improving image quality according to an embodiment of the present invention.
- FIG. 10 is a flowchart illustrating a process of generating a specialized neural network file based on additional learning according to an embodiment of the present invention.
- a resolution improving method includes processing a video data in a server providing streaming video data, acquiring grid generation pattern information based on the processed video data, and applying the grid generation pattern information to the grid generation pattern information.
- FIG. 1 is a diagram illustrating a configuration of a system for performing resolution improvement according to an exemplary embodiment of the present invention.
- a resolution improvement system may include a server 100 and a user device 200.
- the server 100 may include a server that provides a VOD service to the user device 200.
- the server 100 may transmit video data to provide a VOD service to the user device 200.
- the server 100 may transmit the same video content or a downscaling file having a reduced resolution to the user device 200, not the original video data.
- the server 100 calculates a neural network file, which is a file required to restore the resolution of video data (downscaling file) to a predetermined level or higher than the original data, and calculates the neural network file to the user device 200. Can be sent together.
- the user device 200 may improve the resolution of the low quality video data (downscaling file) provided from the server 100 based on the neural network file.
- the user device 200 may select video data to be transmitted (eg, select a content name) and request streaming or download from the server 100.
- the user device 200 may calculate the user viewing pattern information calculated based on the selection information and the reproduction information of the video data of the user device 200 and transmit the calculated user viewing pattern information to the server 200.
- FIG. 2 is a diagram illustrating an example of an image quality improvement operation according to an exemplary embodiment of the present invention.
- the user device 200 may generate a video file having improved resolution through a neural network file.
- the neural network file according to an embodiment of the present invention may be combined with any video file transmitted to the user device 200 to improve the resolution.
- the video file transmitted for the purpose of streaming or downloading from the server 100 to the user device 200 may have a form in which one piece of content is divided into pieces, as shown in FIG. 2.
- the neural network file may also be divided corresponding to each divided video file.
- Each neural network file and video file may be labeled to be combined within the user device 200 after being transmitted from the server 100 to the user device 200, respectively.
- the division of the video file and the division of the neural network file in the user device 200 may be matched with corresponding items, respectively, to improve the resolution of the video file.
- the neural network file includes data related to an artificial neural network algorithm for restoring the resolution of a video file. Accordingly, the user device 200 performs an artificial neural network calculation process based on neural network file divisions for each video file segment. To restore the resolution.
- the video file of the present invention may be a downscaling file obtained by artificially converting original video data held by a server to low quality, or may be original video data having a resolution less than or equal to a reference value.
- FIG. 3 is a block diagram illustrating a configuration of a server according to an exemplary embodiment of the present invention.
- the server 100 may include a communication unit 110, a storage unit 120, and a controller 130.
- the controller 130 may include a data processor 131, a neural network learner 132, and a result evaluator 133.
- the communication unit 110 may use a network for data transmission and reception between a user device and a server, and the type of the network is not particularly limited.
- the network may be, for example, an IP (Internet Protocol) network providing a transmission / reception service of a large amount of data through an Internet protocol (IP), or an All IP network integrating different IP networks.
- IP Internet Protocol
- the network may include a wired network, a wireless broadband network, a mobile communication network including WCDMA, a high speed downlink packet access (HSDPA) network, and a long term evolution (LTE) network, LTE advanced (LTE-A). ), Or one of a mobile communication network including 5G (Five Generation), a satellite communication network, and a Wi-Fi network, or a combination of at least one of them.
- 5G Wireless Generation
- satellite communication network and a Wi-Fi network
- the communication unit 110 may perform data communication with an external web server and a plurality of user devices.
- the communication unit 110 may receive content data (pictures and videos) including images from other web servers or user devices (including administrator devices).
- the server 100 includes a server providing a VOD service, the server 100 may transmit VOD content corresponding to a user request to the user device 200.
- the communication unit 110 may receive and transmit a VOD file for a VOD service, but is not limited thereto.
- the communication unit 110 collects learning data required to generate a neural network file for resolution improvement.
- a communication function can be performed.
- the neural network file may include information necessary for reconstructing the resolution of damaged image data similarly to the original data through an artificial neural network algorithm, and may include information about various parameters to be selected when the artificial neural network algorithm is driven. have.
- the storage unit 120 may include, for example, an internal memory or an external memory.
- the internal memory may be, for example, volatile memory (for example, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM), etc.), non-volatile memory (for example, OTPROM (one). time programmable ROM (PROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash), hard drives, Or it may include at least one of a solid state drive (SSD).
- volatile memory for example, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM), etc.
- non-volatile memory for example, OTPROM (one).
- the external memory may be a flash drive such as compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (mini-SD), extreme digital (XD), It may further include a multi-media card (MMC) or a memory stick.
- the external memory may be functionally and / or physically connected to the electronic device through various interfaces.
- the storage unit 120 reduces the processed data (original image data) by processing image data (eg, photo and video data) received from a user device (administrator device) or another web server at a predetermined ratio. Data or data that has been reduced and then enlarged to the original data size) and the corresponding original data can be matched and stored.
- the original data and the processed data may be used to extract information on the lattice phenomenon generated when the resolution is reduced, respectively.
- the storage unit 120 may store a neural network file that is data for restoring the resolution to the original image level by removing a grid existing in the processed data through an artificial intelligence algorithm (eg, SRCNN) after extracting information about the grid phenomenon.
- an artificial intelligence algorithm eg, SRCNN
- the controller 130 may also be referred to as a processor, a controller, a microcontroller, a microprocessor, a microcomputer, or the like.
- the control unit may be implemented by hardware (hardware) or firmware (firmware), software, or a combination thereof.
- an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above.
- the software code may be stored in a memory and driven by the controller.
- the memory may be located inside or outside the user terminal and the server, and may exchange data with the controller by various known means.
- the controller 130 may generate a neural network file which is a file required to improve the resolution of image data through an artificial neural network based operation.
- the controller 130 may include a data processor 131, a neural network learner 132, a result evaluator 133, and a usage pattern calculator 134.
- the data processor 131 may collect and process the training data necessary to calculate a neural network file required for improving the quality of video data.
- the data processing unit 131 may perform the first change (reduction change) and the second change (expand the reduced image data again) of the collected data. More detailed contents of the operation performed by the data processing unit 131 will be described with reference to FIG. 4.
- the neural network learning unit 132 may perform an neural network learning operation based on artificial intelligence based on the processed data calculated after collecting and processing the data performed by the data processing unit 131.
- the neural network learner 132 may perform a parameter setting operation and a neural network file calculation operation required for a learning process. A more detailed description of the neural network learner 132 will be described with reference to FIG. 5.
- the result evaluator 133 may perform an operation of evaluating a result value of applying the neural network file calculated by the neural network learner 132 in the user device 200.
- the result evaluator 133 may determine the degree of improvement of the resolution of the data as a result of applying the neural network file in the user device 200. In addition, the result evaluator 133 may determine an error rate between the result data and the original data after applying the neural network file.
- the comparison unit of the result data and the original data may be each frame constituting the image, or may be a fragmentation unit divided for transmission of the image.
- each image data is a plurality of frame bundles (for example, when one image is displayed over 100 frames, the 100 frames may be set as one frame bundle, based on image identity). It may also be divided into a plurality). Accordingly, the comparison unit set to compare the result data with the original data may be a divided unit based on image identity.
- the result evaluator 133 may request correction of weight, bias values, etc. constituting the neural network file. That is, the result evaluator 133 may determine whether to modify the parameter constituting the neural network file by comparing the original data with the result data.
- the result evaluator 133 may calculate the importance of each image object required to understand the image from the original data.
- the error rate of one unit (e.g., one frame, one frame bundle, etc.) of the original data and the result data (data of which the resolution is applied by applying a neural network file in the user device 200) is equal to or greater than a preset value. If it is determined that an image object having a predetermined value or more is included, the weight and bias values of the neural network file may be requested to be corrected.
- each image object may be calculated based on a size ratio occupied by one image object in one frame, a repetition ratio of the image object, and the like.
- the result evaluator 133 may calculate the importance of each image object based on the content characteristic of the video data according to various embodiments.
- the result evaluator 133 may first check the content characteristic of the video data.
- the content characteristic of the video video data may be calculated by the data processor 131.
- the content characteristic of the video data may include information about a path where the video data is uploaded to the server 100 (for example, a folder name selected by the user or an administrator when uploading the video file to the server 100), or a user or administrator. It may be calculated based on the content genre, field, and the like inputted when the video file is uploaded to the server 100.
- the content characteristic of the calculated video data may be managed as metadata of the corresponding video data.
- the result evaluator 133 may check the content characteristic information of each video data extracted and stored at the time of uploading to the server 100, and may calculate the importance for each image object based on the content characteristic information. For example, the result evaluator 133 may classify items of the image object into face objects, text objects (eg, subtitles), object objects, etc. of persons, and may determine each object item matching the content characteristic information. In detail, an item having a high importance for each image object in drama content may be set as a face object of a person, and a lecture item may have a high importance for a text item.
- FIG. 4 is a diagram illustrating a configuration of a data processing unit according to an exemplary embodiment of the present invention.
- the data processor 131 may include a size changer 131a, an image divider 131b, and a feature region extractor 131c.
- input data to be input to an input layer or characteristic values of input data to be prepared in the neural network learning unit must be prepared. You can prepare the data.
- the size changing unit 131a performs a first change operation of reducing the size of an image constituting video data by a preset value from the original size and a second change operation of expanding the first change result image corresponding to the original size. can do.
- the secondary change operation may be selectively performed.
- FIG. 6 Reference will be made to FIG. 6 to describe a size changing operation performed by the size changing unit 131a.
- FIG. 6 is a diagram illustrating an example of a size change operation performed by a size change unit according to an exemplary embodiment of the present invention.
- the size changing unit 131a may perform operation a, which is a primary changing operation of reducing the original image 605 by a predetermined ratio, and the reduced size 610 calculated as a result of operation a has the same size as the original image.
- B operation which is a secondary change operation that is enlarged to.
- the image 615 generated after the machining operation (primary change operation (a) and secondary change operation (b)) has a lower resolution than the original image 605, which means that the pixels constituting the image are enlarged in size without increasing numerical values. Is due to
- the server 100 may perform neural network training based on the processed image 615 and the original image 605 to convert the resolution level from the downscaled downscaled image 615 to the original image 605.
- the size changing unit 131a of the data processing unit 131 performs a first change operation to reduce the size of the original image by a predetermined value and a second enlargement of the reduced image generated by the first change operation to the same size as the original image.
- the change operation can be performed.
- the data processing unit 131 may perform an operation of extracting the original image and the processed image generated by the first and second change operations as training data.
- the data processing unit 131 before performing neural network learning, extracts pattern information (location information, color information, etc.) of a grid generated in a processed image (an image which is enlarged after size reduction) and learns data about the neural network. It can be used as input data for.
- pattern information location information, color information, etc.
- the image divider 131b may divide the video data held by the server 100 based on a preset condition. In this case, the image divider 131b may perform an operation of dividing the video data based on the number of frames. Alternatively, the image divider 131b may divide the video data into bundles (chunks) with frames having a matching rate equal to or greater than a preset reference value (eg, 90%) based on the matching rate of the image object. For example, in the division unit, when the same person is photographed, the image division unit 131b may divide the video data into a plurality of chunks for each unit delivered to the user device 200 when the server 100 provides a streaming service to the user device 200.
- a preset reference value eg, 90%
- the chunks divided by the image divider 131b may be used when evaluating AI neural network learning and resolution improvement results.
- the feature region extractor 131c may perform an operation of extracting a feature region including a feature image based on each frame or division unit of video data.
- the feature region extractor 131c may determine whether an image region having a predetermined feature region requirement exists in each frame or division unit.
- the feature region may be determined according to whether an image object having an importance of an image object corresponding to a content field (genre) is greater than or equal to a preset value.
- the feature region extractor 131c may set an image object importance high for the face image of the main character in the drama content, and thus the feature region may be a region (eg, a nook area; a background where the face image of the main character is displayed). Object display area).
- the feature region extractor 131c may perform an operation of extracting not only the feature region within the image but also a specific frame or a specific division unit among all frames or images of the video data.
- a learning importance weight may be assigned to increase the number of learning repetitions.
- the feature region extracted by the feature region extractor 131c may be requested to generate an increased number of processing data. For example, assuming that there is a region set as a feature region in one frame and b region set as a general region, size reduction of the increased number of times (eg, 5 times) in case of a region (eg, 80%, 50%) , 30%, 20%, and 10%) can be used to generate five processed images, and in the case of b area, two processed images can be generated by reducing the size of a normal number (eg, two times). As a result, the feature region selected by the feature region extractor 131c may have a higher resolution reconstruction accuracy than the normal region.
- FIG. 5 is a diagram illustrating a configuration of a neural network learning unit according to an embodiment of the present invention.
- the neural network learner 132 may include a learning importance check unit 132a, a similar data learning support unit 132b, and a neural network file calculator 132c.
- the neural network learner 132 may perform a deep learning learning process based on an artificial neural network, and thus generate a neural network file that is a file required for improving image quality of low resolution video data.
- FIG. 7 is a view for explaining an example of a deep learning learning operation according to an embodiment of the present invention.
- a perceptron that is a neural network model including an input layer, a hidden layer, and an output layer is disclosed.
- the neural network learning disclosed herein may be performed using a multilayer perceptron implemented to include at least one hidden layer.
- the perceptron can input a plurality of signals and output one signal.
- the weight and bias required in the calculation process using the artificial neural network model can be calculated as appropriate values through the backpropagation method.
- appropriate weight data and bias data are extracted through such backpropagation.
- the neural network file calculated through the artificial neural network to perform the resolution improvement may include information about the extracted appropriate weight data and bias data.
- the neural network learner 132 can perform learning using a CNN (Convolution Neural Network) model among artificial neural network models.
- CNN Convolution Neural Network
- there are features such as maintaining the shape of input / output data of each layer, effectively recognizing features with adjacent images while maintaining spatial information of the images, and extracting and learning features of images with a plurality of filters. .
- the basic operation of CNN can be a method of learning a single image through a filter by scanning a partial region one by one and finding a value for it. At this time, the goal of CNN is to find a filter with appropriate weight.
- filters can generally be defined as square matrices such as (4,4) or (3,3).
- the setting value of the CNN filter according to the embodiment of the present invention is not limited.
- the filter may calculate the composite product while iterating over the input data at specified intervals.
- the learning importance checking unit 132a may check the learning importance given to the feature region or the specific frame bundle of the learning data.
- the feature region extractor 131c may divide the frame into a plurality of regions and set at least one of the regions as the feature region.
- the reference for dividing one frame is not limited.
- the feature region extractor 131c may assign different learning importance to each feature region according to a reference element (eg, importance for each image object, size, etc.) in which the feature region is set.
- the learning importance checking unit 132a may check the learning importance included in the learning data after receiving the learning data from the data processing unit 131.
- the training data received from the data processor 131 may be identified in a plurality of divided forms, and accordingly, the learning importance checking unit 132a may check the learning importance assigned to each divided unit.
- the learning data received from the data processing unit 131 may be in an undivided state. Accordingly, the learning importance checking unit 132a may check the learning importance assigned to each frame.
- the learning importance checking unit 132a may check the learning importance for each region in the frame.
- the learning importance checking unit 132a may extract learning option information such as the number of learning that the learning importance imparted per division unit or frame and whether learning is performed through similar data. For example, an operation of extracting option information meaning learning importance may extract option information indicating that the learning count is 3 times and that learning through similar data is not performed. In this case, the number of learning is four times, and may be a method of extracting option information of performing learning through similar data.
- the learning importance checking unit 132a may transmit a command based on the option information indicating the learning importance to the similar data learning supporting unit 132b and the neural network file calculating unit 132c.
- the similar data learning support unit 132b may support to perform learning through similar data.
- the learning importance checking unit 132a checks the option information corresponding to the learning importance, and it is determined that learning through similar data is performed in the option information, the learning importance checking unit 132a calculates the similar data learning support unit 132b and the neural network file.
- the relevant command may be transmitted to the unit 132c.
- the similar data learning support unit 132b may perform an operation of obtaining similar data similar to the target image.
- the similar data may mean similar images searched through an external web.
- the similar data learning support unit 132b may perform an operation of searching for and acquiring images of the cosmos through a portal web. have.
- the similar data learning support unit 132b may select and acquire similar data based on the similarity of the number of objects, the resolution, the similarity of color combinations, and the like among the searched images.
- the neural network file calculator 132c may set an initial parameter value for performing a learning process on image data through a CNN.
- the neural network file calculator 132c may determine a frame size of the original data, a reduction ratio set in the original data when the processed data is generated, and set initial parameters corresponding thereto.
- the neural network file calculator 132c may specify a type of image data required for artificial neural network learning and request to input the corresponding image data as training data.
- the neural network file calculating unit 132c may further include frame information including related image objects in order to perform repetitive learning on a main person, in case of content having a high proportion of persons such as dramas or movies, to the similar data learning support unit 132b. You can request a type of image data required for artificial neural network learning and request to input the corresponding image data as training data.
- the neural network file calculating unit 132c may further include frame information including related image objects in order to perform repetitive learning on a main person, in case of content having a high proportion of persons such as dramas or movies, to the similar data learning support unit 132b. You can request
- the neural network file calculating unit 132c may perform an operation of inputting and processing the data processed by the data processing unit 131 into a preset artificial neural network model.
- the neural network file calculating unit 132c inputs the original data and the processed (reduced to a preset ratio) data into the CNN algorithm to provide information (grid generation pattern information) about the grid generated in the process of changing from the original data to the processed data. Can be extracted.
- the grid generation pattern information calculated by the neural network file calculation unit 132c may be calculated based on the difference between the original data and the processed data, and may include the position of the grid and the pattern information about the color change of the grid. have.
- the neural network file calculator 132c may generate a neural network file required to restore an image to original data by erasing a grid from the processed data based on the calculated grid generation pattern information.
- the neural network file calculating unit 132c inputs downscaling (processing data) data into the artificial neural network algorithm as input data, and when the resolution of the output data shows a matching ratio equal to or higher than the preset value, the data is output. You can end the learning process.
- the neural network file calculator 132c may repeatedly input a myriad of different types of processed data into an input layer and repeatedly determine an agreement rate with the original data as a result of artificial neural network calculation.
- the neural network file calculator 132c may calculate grid generation pattern information generated when the image of a specific size is reduced by inputting various kinds of original data and processed data. Accordingly, the neural network file calculator 132c may calculate grid generation pattern information commonly generated when the image is reduced in not only a specific image but also various images.
- the neural network file calculator 132c inputs processed data into an input layer, and when the matching ratio between the output data and the original data is equal to or greater than a preset value, the parameters set in the artificial neural network algorithm (weight, bias, learning rate). Neural network file including information such as layer-specific activation function).
- the user device 200 when the neural network file calculated by the neural network file calculating unit 132c is transmitted to the user device 200, the user device 200 receives the neural network file and performs artificial neural network test based on the information based on the neural network file with low quality video data (downscaling data). Can be performed, and thus the resolution improvement function of the video data can be performed.
- low quality video data downscaling data
- FIG. 8 is a diagram illustrating a configuration of a user device according to an exemplary embodiment of the present invention.
- the user device 200 may include a communication unit 210, a storage unit 220, an input unit 230, a display unit 240, a camera unit 250, and a controller 260.
- the controller 260 may include a video player 261, a resolution converter 262, and a user information collector 263.
- the communication unit 210 may perform a communication function for receiving a neural network file and video data from the server 100. Furthermore, the communication unit 210 may perform a communication operation for transmitting the feedback information collected by the user device 200 to the server 100.
- the storage unit 220 may store neural network files and video data received from the server 100 according to an embodiment of the present invention. According to various embodiments of the present disclosure, the storage unit 220 stores or temporarily stores the result data (data of which resolution is improved), which is a result of driving an artificial neural network algorithm operation by applying a neural network file to downscaling data having a resolution lower than a preset reference value. Can be.
- the storage unit 220 may store the generated feedback information.
- the storage 220 may store information required for calculating feedback information. For example, when one frame is extracted to provide feedback information among the result data (resolution-enhanced data) generated as a result of the calculation of the artificial neural network algorithm, the reference information for the extraction (eg, the user's detected during video playback) Content of extracting the frame at the time when facial distortion is detected, and the like.
- the input unit 230 may receive user selection information regarding a content genre, a content name, and the like.
- the display unit 240 may display a playback screen of the corresponding video when the video data received from the server 100 or the result data after the resolution improvement operation of the video data is reproduced.
- the camera unit 250 may take a picture and a video in response to a user request.
- the camera unit 250 may upload image information such as photographed pictures and videos to the server 100 or other web servers. Alternatively, the image information photographed through the camera unit 250 may be transmitted to another user device.
- the camera unit 250 may first determine the resolution based on a user request. According to an embodiment of the present disclosure, the camera unit 250 may reduce and store the resolution of a picture or a video to be lowered to or below a preset level based on whether neural network file is installed to improve image quality.
- the camera unit 250 may operate a camera that regularly photographs the user's face at a predetermined reference interval while reproducing the resultant data having improved resolution. It is possible to determine whether a user's facial expression or beauty hair is distorted, and feedback information may be extracted correspondingly.
- the controller 260 may perform resolution conversion and playback of a video file downloaded from the server 100.
- the controller 260 may include a video player 261, a resolution converter 262, and a user information collector 263.
- the video player 261 may control to play a streamed video file and display it on the display unit 240.
- the video reproducing unit 261 may determine the resolution of the video data requested for output. If it is determined that the resolution of the requested video data resolution is required to be lower than a preset level, the video reproducing unit 261 may request the resolution conversion unit 262 to improve the resolution. Thereafter, the video player 261 may play a file having an improved resolution through the resolution converter 262.
- the resolution converter 262 may determine a resolution level of current image data (pictures and videos) and a target resolution requested by the user. At this time, the resolution converting unit 262 matches the divided video data received from the server 100 and the divided neural network file, and then runs an artificial neural network algorithm to convert the downscaling data to a desired level of resolution. Can be.
- the user information collecting unit 263 may collect user information for feedback.
- the user information collecting unit 263 may select and store a frame to be used as feedback information among result data after the resolution is improved based on an artificial neural network algorithm.
- the user information collecting unit 263 may acquire the face information of the user while the user plays the video data with improved resolution, and when an event such as frowning of the user occurs, at the time when the event occurs
- the video frame information being displayed can be collected.
- the user information collection unit 263 may collect content information, such as an item, a genre, and the like of content that is reproduced above a reference value. For example, the user information collection unit 263 may determine the reproduction frequency of the animation compared to the documentary (photo image based), the reproduction frequency of the non-subtitle content compared to the content in which the subtitle exists, and collect information on the same.
- the reproduction information collected by the user information collecting unit 263 may be provided to the server 100, and the server 100 may calculate user pattern information based on the reproduction information.
- FIG. 9 is a diagram illustrating a generation and transmission process of an image file for improving image quality according to an embodiment of the present invention.
- the server 100 may generate a neural network file for improving resolution and transmit the file to the user device 200.
- the server 100 may perform operation 705 for processing video data in a preset data set.
- the data set may mean a training data set for generating a basic neural network file.
- the data set consists of arbitrary video data having various genres, various subjects, and various formats, and some meta information including resolution may be standardized. Accordingly, various video data included in the data set may be preprocessed to have the same resolution.
- Operation 705 is an operation for generating data for training the artificial neural network algorithm, and may perform downscaling processing to reduce the resolution of video data in order to generate data suitable for learning.
- operation 705 may be a processing operation (image reduction, downscaling) for each frame constituting the video file.
- operation 705 may be an operation of selecting a frame to be input for artificial neural network learning through sampling of each division unit of the video file, and then processing (downscaling the resolution by a preset ratio) with respect to the corresponding frame. For example, if a video file has a total of 2400 frames, assuming that 24 frames consist of 100 chunks in one unit, the server 100 samples one frame per video segmentation unit and uses a total of 100 frames as training data. I can process it.
- the server 100 may perform operation 710 for obtaining processed video data based grid generation pattern information.
- the processed video data may refer to data obtained by reducing the size of the original data (data designated as learning source data only for data having a preset resolution level or more) at a preset ratio.
- the server 100 may obtain pattern information for generating the grid by comparing the processed image in which the grid phenomenon is generated and the original image.
- the obtained grid generation pattern information may be used to restore the resolution by removing the grid from the image in which the grid phenomenon is generated later.
- the server 100 may perform operation 715 of generating a neural network file for improving image quality based on the grid generation pattern information.
- the server 100 generates a basic neural network file by calculating artificial neural network algorithm information (activation function applied by layer, weight, bias, etc.) required to remove the grid from the downscaled image data generated by the grid and restore the original image. Can be.
- Factors such as weight and bias provided as a result value may be determined based on the matching ratio between the final result (quality improvement data) and the original image data.
- the server 100 may determine weight and bias information applied to the neural network operation as information to be included in the neural network file.
- the server 100 may perform operation 720 to confirm that the first streaming request (or download request) for the video data is received from the user device 200.
- the server 100 may perform operation 725 for transmitting a low quality version (downscaling data) of the requested video data together with the basic neural network file for improving image quality generated in the user device 200. Accordingly, since the user device 200 receives the low quality version (downscaling data) of the video, the user device 200 can easily receive the content without the restriction of the network environment, and the basic neural network file received together with the received low quality version of the video data ( By applying to downscaling data, users can play high-definition video of the desired level.
- FIG. 10 is a flowchart illustrating a process of generating a specialized neural network file based on additional learning according to an embodiment of the present invention.
- the controller 130 of the server 100 may acquire new video data and perform operation 805 to confirm acquisition of data. Thereafter, the controller 130 may perform operation 810 to identify an additional learning condition of the new video data. For example, the controller 130 may determine whether a result of performing the restoration operation based on the basic neural network file generated in operation 715 of FIG. 9 shows a restoration rate of more than a reference value, and thus determines whether to perform additional learning. Can be. In this case, the determination of the reconstruction rate may be performed based on values of SSIM (Structural Similarity) and PSNR (Peak Signal-to-Noise Ratio).
- SSIM Structuretural Similarity
- PSNR Peak Signal-to-Noise Ratio
- the controller 130 when it is determined that the newly acquired video data satisfies the additional learning condition (eg, less than 0.90 SSIM and less than 30 PSNR), the controller 130 performs additional learning on new video data. Can be performed.
- the additional learning may mean performing specialized image learning on one item of new video data.
- the controller 130 in order to perform additional learning, the controller 130 may adjust a meta value including the resolution of new video data according to a specification in previously performed image learning. As the preprocessing operation of identifying the standard with the data set used when generating the basic neural network file is completed, additional image learning may be performed.
- the controller 130 may perform operation 820 for generating a specialized neural network file for new video data.
- the specialized neural network file may be generated according to a result of additionally performing artificial neural network learning on new video data added after applying an artificial neural network algorithm and parameters based on the basic neural network file as initial values. That is, in order to generate the specialized neural network file, the operation of loading the basic neural network file generated in operation 715 of FIG. 7 may be essentially preceded.
- the controller 130 may perform operation 830 for transmitting a downscaling file and the specialized neural network file to the user device 200.
- the second streaming request may be a request for a specialized neural network file to restore the resolution of video data.
- the division of the first streaming request and the second streaming request may be divided based on the type of service used by the user. For example, a request received from a user using a low-cost plan is a first streaming request, which is a streaming scheme that provides a basic neural network file, while a streaming request received from a user using a relatively expensive plan is a specialized neural network file. It may be a second streaming request that is a method of transmitting a.
- the user device 200 may transmit feedback information on the state of the video data, which has been played back or converted, to the server 100. Accordingly, the user device 200 may calculate playback-related information for each user, such as a content genre, a content feature, and a main playback request time zone, which are reproduced at a frequency greater than or equal to a reference value, and may transmit the same to the server 100.
- playback-related information such as a content genre, a content feature, and a main playback request time zone, which are reproduced at a frequency greater than or equal to a reference value
- the user device 200 may provide a frame sample of the result data after the resolution improvement operation is completed to the server 100 according to a preset period. Accordingly, the server 100 may compare the result data frame calculated after the resolution improvement operation received from the user device 200 with the original data frame of the same content.
- the transmitted frame information may be transmitted along with the reproduction position information of the content. Accordingly, the server 100 may search for a frame image to be compared in the original data.
- the server 100 may compare an image frame provided for feedback from the user device 200 with an original image frame of the corresponding content, and determine a matching rate. If it is determined that the matching rate is less than or equal to the preset reference value, a re-learning operation for updating the neural network file may be requested, and the re-learning operation may be performed accordingly.
- neural network files generated according to various embodiments of the present invention may be compressed as necessary.
- the server 100 may compress the neural network file in consideration of the performance of the user device 200, and transmit the compressed neural network file to the user device 200.
- the neural network file may be compressed using at least one of pruning, quantization, decomposition, and knowledge distillation. Pruning is one of compression techniques to delete weight and bias among neural network file weights and biases or to have little effect on output values. Quantization is one of compression techniques that quantize each weight to a predetermined bit. Decomposition is one of compression techniques that reduce the size of weight by linearly decomposition the weight matrix or tensor, a set of weights. Knowledge Distillation is one of the compression techniques that create and train a Student model smaller than the original model using the original model as the Teacher model. At this time, the Student model may be generated through the above-described Pruning, Decomposition or Quantization.
- the degree of compression according to the performance of the user device 200 may be determined through various methods.
- the degree of compression of the neural network file may be determined based on a simple specification of the user device 200. That is, the degree of compression may be collectively determined by the specifications and memory specifications of the processor of the user device 200.
- the degree of compression of the neural network file may be determined based on the usage state of the user device 200.
- the server 100 may receive usage state information from the user device 200 and obtain available resource information of the user device 200 according to the received usage state information.
- the server 100 may determine the degree of compression of the neural network file based on the available resource information of the user device 200.
- the available resource information may refer to information on an application being executed by the user device 200, a CPU or GPU occupancy rate determined according to an application being executed, and information related to a memory capacity that can be stored in the user device 200.
- the resolution improving method in the server providing the video data for streaming, processing the video data, obtaining grid generation pattern information based on the processed video data, grid generation pattern A generation step of generating a neural network file required to improve the resolution of the video data based on the information; when receiving a streaming request from a user device, the requested video data and the resolution of the requested video data are restored And a transmission step of dividing a neural network file and transmitting the file to the user device.
- the generating step is a basic neural network file generation step of generating a basic neural network file based on a plurality of video data included in a preset data set, and when it is determined that the acquired new video data satisfies an additional learning condition.
- the method may include generating a specialized neural network file corresponding to the new video data.
- the additional learning step may further include adding a downscaling file of the new video data according to values of SSIM (Structural Similarity) and Peak Signal-to-Noise Ration (PSNR) of a result of performing a resolution reconstruction operation based on the basic neural network file.
- the method may include determining whether the learning condition is satisfied.
- the processing step is to divide the video data, based on the matching rate of the image object, a plurality of frames having a matching rate equal to or more than a predetermined reference value in a chunk, divided into a plurality of chunks and the size of the image constituting the video data
- the method may include a size changing step of performing a first changing operation of reducing the original size from the original size by a predetermined value, and optionally performing a second changing operation of expanding the first changed image to the original size.
- the processing may include extracting a feature region including a feature image based on each frame or division unit of the video data and assigning learning importance to the extracted feature region.
- the feature area may be an area including an image object having an importance of an image object corresponding to a content field more than a preset value.
- the generation step may include a learning importance checking step of checking a learning importance given to a feature region or a specific frame bundle of the learning video data, extracting option information representing the learning importance, and original data of an original size from the learning video data.
- the processing data which is reduced data at a predetermined ratio, is input to the CNN algorithm for training, and the matching rate between the calculation result value obtained by inputting the processing data into the artificial neural network and the original data is determined, and the matching rate is equal to or greater than the preset value.
- the option information may include information on whether the learning is performed through the number of learning and similar data.
- the generation step when learning the learning data set the learning importance, the operation to obtain similar data similar to the learning target image as the learning execution command using similar data is confirmed, but the similarity of the resolution and color combination It may include a similar data acquisition step of obtaining similar data based on.
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
Claims (8)
- 스트리밍용 비디오 데이터를 제공하는 서버에서, 비디오 데이터를 가공하는 가공 단계; A processing step of processing the video data in a server providing the video data for streaming;상기 가공된 비디오 데이터에 기반하여 격자 생성 패턴 정보를 획득하고, 격자 생성 패턴 정보에 기반하여 상기 비디오 데이터의 해상도를 개선하기 위해 요구되는 신경망 파일을 생성하는 생성 단계; A generation step of obtaining grid generation pattern information based on the processed video data and generating neural network files required to improve the resolution of the video data based on grid generation pattern information;사용자 기기로부터 스트리밍 요청을 수신하면, 요청된 비디오 데이터및 상기 요청된 비디오 데이터의 해상도를 복원하는 데 요구되는 신경망 파일을 분할하여 상기 사용자 기기로 전송하는 전송 단계;를 포함하는 것을 특징으로 하는 해상도 개선 방법.Receiving a streaming request from a user device, and transmitting the divided video data and the neural network file required to restore the resolution of the requested video data to the user device. Way.
- 제 1항에 있어서, The method of claim 1,상기 생성 단계는The generating step기 설정된 데이터 셋에 포함된 다수의 비디오 데이터를 기반으로 기본 신경망 파일을 생성하는 기본 신경망 파일 생성 단계; A basic neural network file generation step of generating a basic neural network file based on a plurality of video data included in a preset data set;획득된 임의의 신규 비디오 데이터가 추가 학습 조건을 만족하는 것으로 판단되는 경우, 상기 신규 비디오 데이터에 대한 추가학습을 수행하되, 상기 기본 신경망 파일을 적용한 인공신경망 알고리즘을 통해 추가학습을 수행하는 추가 학습 단계; If it is determined that the acquired new video data satisfies the additional learning condition, additional learning is performed on the new video data, but additional learning is performed through an artificial neural network algorithm applying the basic neural network file. ;상기 추가 학습의 결과로 상기 신규 비디오 데이터의 다운스케일링 파일과, 상기 신규 비디오 데이터에 대응하는 특화 신경망 파일을 생성하는 특화 신경망 파일 생성 단계;를 포함하는 것을 특징으로 하는 해상도 개선 방법And generating a downscaling file of the new video data and a specialized neural network file corresponding to the new video data as a result of the additional learning.
- 제 2항에 있어서,The method of claim 2,상기 추가 학습 단계는 The additional learning step상기 신규 비디오 데이터의 다운스케일링 파일을 상기 기본 신경망 파일에 기반한 해상도 복원 동작을 수행한 결과물의 SSIM(Structural Similarity), PSNR(Peak Signal-to-Noise Ration)의 값에 따라 추가 학습 조건을 만족하는지 여부를 판단하는 단계를 포함하는 것을 특징으로 하는 해상도 개선 방법.Whether the downscaling file of the new video data satisfies additional learning conditions according to values of SSIM (Structural Similarity) and PSNR (Peak Signal-to-Noise Ration) of the result of performing the resolution reconstruction operation based on the basic neural network file. Resolution improvement method comprising the step of determining.
- 제 1항에 있어서,The method of claim 1,상기 가공 단계는 The processing step비디오 데이터를 분할하되, 이미지 객체의 일치율에 기반하여 일치율이 기 설정된 기준치 이상인 다수의 프레임들을 하나의 청크로 묶어, 다수개의 청크로 분할하는 분할 단계; 및A segmentation step of dividing the video data, grouping a plurality of frames whose match rate is equal to or greater than a predetermined reference value based on the match rate of the image object into one chunk, and dividing the video data into a plurality of chunks; And비디오 데이터를 구성하는 이미지의 사이즈를 원본 사이즈로부터 기 설정된 값만큼 축소시키는 1차 변경 동작을 수행하며, 선택적으로 1차 변경된 이미지를 원본 사이즈로 확대시키는 2차 변경 동작을 수행하는 사이즈 변경 단계;를 포함하는 것을 특징으로 하는 해상도 개선 방법.A size changing step of performing a first changing operation of reducing the size of an image constituting video data by a predetermined value from an original size, and optionally performing a second changing operation of expanding a first changed image to an original size; Resolution improvement method comprising the.
- 제 3항에 있어서,The method of claim 3, wherein상기 가공 단계는 The processing step비디오 데이터의 각 프레임 또는 분할 단위를 기준으로 특징 이미지가 포함된 특징 영역을 추출하고, 추출된 특징 영역에 대하여 학습 중요도를 부여하는 특징 영역 추출 단계를 포함하고, Extracting a feature region including a feature image based on each frame or division unit of video data, and extracting a feature region to give a learning importance to the extracted feature region;상기 특징 영역은 콘텐츠 분야에 대응하는 이미지 객체 중요도가 기 설정값 이상인 이미지 객체를 포함하는 영역인 것을 특징으로 하는 해상도 개선 방법.And the feature area is an area including an image object having an importance of an image object corresponding to a content field equal to or greater than a predetermined value.
- 제 1항에 있어서, The method of claim 1,상기 생성 단계는 The generating step학습용 비디오 데이터의 특징 영역 또는 특정 프레임 묶음에 대하여 부여된 학습 중요도를 확인하고, 상기 학습 중요도가 의미하는 옵션 정보를 추출하는 학습 중요도 확인 단계; 및A learning importance checking step of checking learning importance given to a feature region or a specific frame bundle of learning video data and extracting option information meaning the learning importance; And학습용 비디오 데이터 중 원본 사이즈인 원본 데이터와, 기 설정된 비율로 축소된 데이터인 가공 데이터를 CNN 알고리즘에 투입하여 학습시키되, Of the training video data, the original data of the original size and the processed data, which is data reduced in a predetermined ratio, are put into the CNN algorithm to be trained.인공 신경망에 상기 가공 데이터를 입력하여 얻은 연산 결과값와 상기 원본 데이터와의 일치율을 판단하여, 일치율이 기 설정된 값 이상이 되게 하는 인공 신경망의 파라미터 및 활성화 함수를 포함하는 신경망 파일을 생성하는 신경망 파일 산출 단계;를 포함하고, Calculate neural network file to generate neural network file including parameters and activation function of artificial neural network to determine the coincidence rate between the calculation result value and the original data obtained by inputting the processing data into artificial neural network Comprising;상기 옵션 정보는 학습 횟수 및 유사 데이터를 통한 학습 수행 여부에 관한 정보를 포함하는 것을 특징으로 하는 해상도 개선 방법.The option information includes the information on the number of learning and whether or not to perform the learning through similar data.
- 제 6항에 있어서, The method of claim 6,상기 생성 단계는 The generating step학습 중요도가 설정된 학습용 데이터를 학습할 시, 유사 데이터를 이용한 학습 수행 명령이 확인됨에 따라 학습 대상 이미지와 유사한 유사 데이터를 획득하는 동작을 수행하되, 해상도 및 색상 조합의 유사도에 기반하여 유사 데이터를 획득하는 유사 데이터 획득 단계;를 포함하는 것을 특징으로 하는 해상도 개선 방법.When learning the learning data for which learning importance is set, the method of acquiring similar data similar to the subject image is acquired according to the learning execution command using similar data, but obtaining similar data based on the similarity of resolution and color combination. Obtaining similar data; step of improving the resolution comprising a.
- 스트리밍용 비디오 데이터를 제공하는 서버에서, 비디오 데이터를 가공하는 가공 단계; A processing step of processing the video data in a server providing the video data for streaming;상기 가공된 비디오 데이터에 기반하여 격자 생성 패턴 정보를 획득하고, 격자 생성 패턴 정보에 기반하여 상기 비디오 데이터의 해상도를 개선하기 위해 요구되는 신경망 파일을 생성하는 생성 단계; A generation step of obtaining grid generation pattern information based on the processed video data and generating neural network files required to improve the resolution of the video data based on grid generation pattern information;사용자 기기로부터 스트리밍 요청을 수신하면, 요청된 비디오 데이터 및 상기 요청된 비디오 데이터의 해상도를 복원하는 데 요구되는 신경망 파일을 분할하여 상기 사용자 기기로 전송하는 전송 단계;및Receiving a streaming request from a user device, and transmitting the divided video data and the neural network file required to restore the resolution of the requested video data to the user device; and상기 사용자 기기로 전송된 분할된 비디오 데이터와, 상기 비디오 데이터의 해상도를 복원하는 데 요구되는 신경망 파일을 매칭하고, 상기 분할된 비디오 데이터를 상기 매칭된 신경망 파일로 인공 신경망 알고리즘 연산을 수행하여, 상기 분할된 비디오 데이터의 해상도를 복원하는 단계;를 포함하는 것을 특징으로 하는 해상도 개선 방법.Matching the divided video data transmitted to the user device with a neural network file required to restore the resolution of the video data, and performing artificial neural network algorithm operation on the divided video data into the matched neural network file; Restoring the resolution of the divided video data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/618,335 US20210160556A1 (en) | 2018-04-24 | 2019-04-23 | Method for enhancing resolution of streaming file |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR20180047305 | 2018-04-24 | ||
KR10-2018-0047305 | 2018-04-24 | ||
KR10-2019-0046954 | 2019-04-22 | ||
KR1020190046954A KR102082816B1 (en) | 2018-04-24 | 2019-04-22 | Method for improving the resolution of streaming files |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019209006A1 true WO2019209006A1 (en) | 2019-10-31 |
Family
ID=68294114
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2019/004891 WO2019209006A1 (en) | 2018-04-24 | 2019-04-23 | Method for improving resolution of streaming files |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2019209006A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114821415A (en) * | 2022-04-24 | 2022-07-29 | 中国水利水电科学研究院 | Video micro-vibration amplification calibration and real-time extraction method based on machine learning |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140072242A1 (en) * | 2012-09-10 | 2014-03-13 | Hao Wei | Method for increasing image resolution |
US20170032222A1 (en) * | 2015-07-30 | 2017-02-02 | Xerox Corporation | Cross-trained convolutional neural networks using multimodal images |
US20170132758A1 (en) * | 2015-11-05 | 2017-05-11 | Facebook, Inc. | Systems and methods for processing content using convolutional neural networks |
US20170347061A1 (en) * | 2015-02-19 | 2017-11-30 | Magic Pony Technology Limited | Machine Learning for Visual Processing |
KR20180003609A (en) * | 2015-05-11 | 2018-01-09 | 넷플릭스, 인크. | Techniques to predict perceptual video quality |
-
2019
- 2019-04-23 WO PCT/KR2019/004891 patent/WO2019209006A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140072242A1 (en) * | 2012-09-10 | 2014-03-13 | Hao Wei | Method for increasing image resolution |
US20170347061A1 (en) * | 2015-02-19 | 2017-11-30 | Magic Pony Technology Limited | Machine Learning for Visual Processing |
KR20180003609A (en) * | 2015-05-11 | 2018-01-09 | 넷플릭스, 인크. | Techniques to predict perceptual video quality |
US20170032222A1 (en) * | 2015-07-30 | 2017-02-02 | Xerox Corporation | Cross-trained convolutional neural networks using multimodal images |
US20170132758A1 (en) * | 2015-11-05 | 2017-05-11 | Facebook, Inc. | Systems and methods for processing content using convolutional neural networks |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114821415A (en) * | 2022-04-24 | 2022-07-29 | 中国水利水电科学研究院 | Video micro-vibration amplification calibration and real-time extraction method based on machine learning |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
KR102082816B1 (en) | Method for improving the resolution of streaming files | |
CN109803180B (en) | Video preview generation method and device, computer equipment and storage medium | |
US20170213086A1 (en) | Cloud platform with multi camera synchronization | |
WO2023071609A1 (en) | Copied image recognition method and related device thereof | |
WO2019209006A1 (en) | Method for improving resolution of streaming files | |
KR102130076B1 (en) | Method for improving the resolution of streaming files based on the learning importance of feature areas | |
WO2019225793A1 (en) | Ai video learning platform-based vod service system | |
WO2019209005A1 (en) | Artificial intelligence based resolution improvement system | |
WO2024125312A1 (en) | Picture quality recognition method, apparatus and device, and storage medium and product | |
DE102012200417B4 (en) | Image data recording device | |
RU2697702C1 (en) | Image compression method, image recovery method, image compression device and image recovery device | |
WO2023080667A1 (en) | Surveillance camera wdr image processing through ai-based object recognition | |
KR20220021495A (en) | Method for improving the resolution of streaming files based on AI | |
WO2022080680A1 (en) | Artificial-intelligence-based image inpainting method and device | |
KR20220021494A (en) | Artificial intelligence based resolution improvement system | |
KR102130078B1 (en) | System for changing AI parameters based on resolution improvement | |
WO2019209007A1 (en) | Ai-based image compression and decompression system | |
KR102130077B1 (en) | System to improve the resolution based on pattern information of grid generation | |
WO2019209008A1 (en) | System for improving video quality by using changed macroblock extraction technique | |
CN109657098B (en) | Video fingerprint extraction method and device | |
CN113973175A (en) | Rapid HDR video reconstruction method | |
KR101997909B1 (en) | Program and recording medium for extracting ai image learning parameters for resolution restoration | |
JP4053251B2 (en) | Image search system and image storage method | |
KR20190140815A (en) | Image learning support system for resolution restoration using user's preference information | |
CN112379856B (en) | Display picture reconstruction device and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19794069 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19794069 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 20.04.2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19794069 Country of ref document: EP Kind code of ref document: A1 |