TW202044199A

TW202044199A - Image processing apparatus and image processing method thereof

Info

Publication number: TW202044199A
Application number: TW108138192A
Authority: TW
Inventors: 林亨俊; 文永秀; 安泰慶
Original assignee: 南韓商三星電子股份有限公司
Priority date: 2019-04-02
Filing date: 2019-10-23
Publication date: 2020-12-01
Also published as: KR102246110B1; KR20200116836A

Abstract

An image processing apparatus and method are provided. The image processing apparatus includes: a memory configured to store at least one instruction, and a processor electrically connected to the memory, wherein the processor, by executing the at least one instruction, is configured to: apply an input image to a training network model; and apply, to a pixel block included in the input image, a texture patch corresponding to the pixel block to obtain an output image, wherein the training network model stores a plurality of texture patches corresponding to a plurality of classes classified based on a characteristic of an image, and is configured to train at least one texture patch, among the plurality of texture patches, based on the input image.

Description

顯示裝置以及其影像處理方法Display device and its image processing method

本揭露是有關於一種影像處理裝置及其影像處理方法，且更具體而言，是有關於一種用於恢復輸入影像的紋理成分的影像處理裝置及其影像處理方法。The present disclosure relates to an image processing device and an image processing method thereof, and more specifically, to an image processing device and an image processing method thereof for restoring texture components of an input image.

本揭露亦是有關於一種人工智慧（artificial intelligence，AI）系統及其應用，所述人工智慧系統使用機器學習演算法來模擬人腦的功能（例如，辨識及判斷）。This disclosure also relates to an artificial intelligence (AI) system and its application. The artificial intelligence system uses machine learning algorithms to simulate the functions of the human brain (for example, identification and judgment).

隨著電子技術的發展，已開發出並廣泛使用各種類型的電子裝置。具體而言，多年來，已開發出用於各種場所（例如，家庭、辦公室及公共場所）的顯示裝置。With the development of electronic technology, various types of electronic devices have been developed and widely used. Specifically, over the years, display devices for various places (for example, homes, offices, and public places) have been developed.

另外，已廣泛推出並使用高解析度顯示面板（例如，4K超高畫質（Ultra High Definition，UHD）電視（television，TV）等）。然而，仍然缺乏高品質的高解析度內容。因此，已利用自低解析度內容產生高解析度內容的各種技術。然而，由於MPEG/H.264/HEVC等的影像壓縮，可能會發生內容的紋理丟失，且因此需要一種用於恢復丟失的紋理成分的技術。In addition, high-resolution display panels have been widely introduced and used (for example, 4K Ultra High Definition (UHD) televisions (television, TV), etc.). However, there is still a lack of high-quality high-resolution content. Therefore, various technologies for generating high-resolution content from low-resolution content have been utilized. However, due to image compression of MPEG/H.264/HEVC, etc., texture loss of content may occur, and therefore a technique for recovering the lost texture components is required.

近年來，實施人類級別的人工智慧（AI）的人工智慧系統已用於各個領域中。與先前技術中基於規則的智慧型系統不同，人工智慧系統是其中機器進行學習、判斷以及變得智慧的系統。人工智慧系統被使用得越多，辨識率越高且對使用者的偏好的理解也越佳。因此，先前技術中基於規則的智慧型系統已逐漸被基於深度學習的人工智慧系統所取代。In recent years, artificial intelligence systems that implement human-level artificial intelligence (AI) have been used in various fields. Different from rule-based intelligent systems in the prior art, artificial intelligence systems are systems in which machines learn, judge, and become intelligent. The more artificial intelligence systems are used, the higher the recognition rate and the better the understanding of user preferences. Therefore, the rule-based intelligent system in the prior art has gradually been replaced by the artificial intelligence system based on deep learning.

人工智慧技術包括機器學習（例如，深度學習）及使用機器學習的元素技術（element technology）。Artificial intelligence technology includes machine learning (for example, deep learning) and element technology using machine learning.

機器學習是一種對輸入資料的特性進行自主分類/訓練的演算法技術。元素技術是一種使用機器學習演算法（例如，深度學習）模擬人腦的功能（例如，辨識及判斷）的技術，且包括語言理解、視覺理解、推理/預測、知識表示（knowledge representation）、運動控制等。Machine learning is an algorithm technology that autonomously classifies/trains the characteristics of input data. Element technology is a technology that uses machine learning algorithms (for example, deep learning) to simulate the functions of the human brain (for example, recognition and judgment), and includes language understanding, visual understanding, reasoning/prediction, knowledge representation, and sports Control etc.

人工智慧技術可應用於各個領域，其實例闡述如下。語言理解是一種用於對人類語言/字符進行辨識及應用/處理的技術，包括自然語言處理、機器翻譯、對話系統、查詢回應、語音辨識/合成等。視覺理解是一種如同人類感知到一樣對對象進行辨識及處理的技術，包括對象辨識、對象跟蹤、影像搜索、人類辨識、場景理解、空間理解、影像增強等。推斷預測是一種用於對資訊進行判斷以及邏輯推斷及預測的技術，包括基於知識/概率的推理、最佳化預測、基於偏好的規劃以及推薦。知識表示是一種用於將人類經驗資訊自動化為知識資料的技術，包括知識構建（資料產生/分類）及知識管理（資料利用）。運動控制是一種用於控制設備（device）或對象的自主運動（例如，車輛的行駛及機器人的運動）的技術，包括運動控制（導航、碰撞及行駛）、操作控制（行為控制）等。Artificial intelligence technology can be applied to various fields, and its examples are described below. Language understanding is a technology used to recognize and apply/process human languages/characters, including natural language processing, machine translation, dialogue systems, query response, speech recognition/synthesis, etc. Visual understanding is a technology that recognizes and processes objects as humans perceive them, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Inference and prediction is a technology used to make judgments and logical inferences and predictions on information, including knowledge/probability-based reasoning, optimized prediction, preference-based planning, and recommendation. Knowledge representation is a technology used to automate human experience information into knowledge data, including knowledge construction (data generation/classification) and knowledge management (data utilization). Motion control is a technology used to control the autonomous motion of a device or object (for example, the movement of a vehicle and the movement of a robot), including motion control (navigation, collision and driving), operation control (behavior control), etc.

先前技術中的影像處理裝置由於應用固定紋理片來恢復丟失的紋理成分，或者應用不太適合於影像的紋理片而存在問題。因此，需要一種用於產生合適於影像的紋理的技術。The image processing device in the prior art has problems due to the application of fixed texture patches to restore the lost texture components, or the application of texture patches that are not suitable for images. Therefore, a technique for generating textures suitable for images is needed.

本揭露提供一種影像處理裝置及其影像處理方法，所述影像處理裝置用於藉由使用基於輸入影像的特性而被訓練的紋理片來增強輸入影像的細節。The present disclosure provides an image processing device and an image processing method thereof. The image processing device is used to enhance the details of the input image by using texture patches trained based on the characteristics of the input image.

附加態樣將在以下說明中予以部分闡述，且所述附加態樣將部分地藉由所述說明而顯而易見，或者可藉由對所提出的實施例進行實踐而得知。Additional aspects will be partly explained in the following description, and the additional aspects will be partly obvious from the description, or can be learned by practicing the proposed embodiments.

根據本揭露的態樣，提供一種影像處理裝置，所述影像處理裝置包括：記憶體，被配置成儲存至少一個指令；以及至少一個處理器，電性連接至所述記憶體，其中所述至少一個處理器藉由執行所述至少一個指令而被配置成：將輸入影像應用至訓練網路模型，以及將與所述輸入影像中所包括的畫素區塊對應的紋理片應用至所述畫素區塊，以獲得輸出影像，其中所述訓練網路模型儲存與基於影像的特性進行分類的多個類別對應的多個紋理片，且被配置成基於所述輸入影像來訓練所述多個紋理片中的至少一個紋理片。According to an aspect of the present disclosure, there is provided an image processing device, the image processing device comprising: a memory configured to store at least one instruction; and at least one processor electrically connected to the memory, wherein the at least A processor is configured by executing the at least one instruction to: apply an input image to a training network model, and apply a texture patch corresponding to the pixel block included in the input image to the image A pixel block to obtain an output image, wherein the training network model stores a plurality of texture patches corresponding to a plurality of categories classified based on the characteristics of the image, and is configured to train the plurality of texture patches based on the input image At least one of the texture pieces.

所述訓練網路模型可被配置成基於所述畫素區塊的特性來辨識所述多個類別中的類別，獲得與所辨識的所述類別對應的紋理片，將所述畫素區塊與所辨識的所述類別之間的第一相似度和所述紋理片與所辨識的所述類別之間的第二相似度進行比較，並基於所述比較來判斷是否更新所述紋理片。The training network model may be configured to identify the categories of the multiple categories based on the characteristics of the pixel blocks, obtain texture patches corresponding to the recognized categories, and convert the pixel blocks Compare with the first similarity between the recognized category and the second similarity between the texture patch and the recognized category, and determine whether to update the texture patch based on the comparison.

所述訓練網路模型可被配置成基於所述比較而以所述畫素區塊來替換與所辨識的所述類別對應的所述紋理片，或者添加所述畫素區塊作為與所辨識的所述類別對應的另一紋理片。The training network model may be configured to replace the texture patch corresponding to the recognized category with the pixel block based on the comparison, or add the pixel block as a reference to the recognized category. Another texture sheet corresponding to the category.

基於根據所述比較得出所述第一相似度小於所述第二相似度，所述訓練網路模型可被配置成保持與所辨識的所述類別對應的所述紋理片；以及基於根據所述比較得出所述第一相似度大於所述第二相似度，所述訓練網路模型可被配置成基於所述畫素區塊更新所述紋理片。Based on the fact that the first degree of similarity is less than the second degree of similarity based on the comparison, the training network model may be configured to maintain the texture patch corresponding to the recognized category; The comparison shows that the first similarity is greater than the second similarity, and the training network model may be configured to update the texture patch based on the pixel block.

基於與所辨識的所述類別對應的所述紋理片包括多於一個紋理片，所述訓練網路模型可被配置成基於所述畫素區塊與所述多於一個紋理片中的每一紋理片之間的關聯來辨識所述多於一個紋理片中的一者。Based on the texture patch corresponding to the identified category including more than one texture patch, the training network model may be configured to be based on each of the pixel block and the more than one texture patch The correlation between texture patches is used to identify one of the more than one texture patches.

所述訓練網路模型可被配置成基於所述至少一個紋理片的儲存時間及所述至少一個紋理片的應用頻率中的至少一者來訓練所述至少一個紋理片。The training network model may be configured to train the at least one texture piece based on at least one of the storage time of the at least one texture piece and the application frequency of the at least one texture piece.

基於根據所述畫素區塊的特性而確定出所述畫素區塊不對應於所述多個類別中的一者，所述訓練網路模型可被配置成基於所述畫素區塊的所述特性而產生新類別，並將所述畫素區塊映射及儲存至所述新類別。Based on determining that the pixel block does not correspond to one of the multiple categories based on the characteristics of the pixel block, the training network model may be configured to be based on the pixel block The characteristics generate a new category, and the pixel blocks are mapped and stored in the new category.

所述訓練網路模型可被配置成辨識與所述輸入影像中所包括的多個畫素區塊中的每一者對應的類別，並基於所述多個類別中的每一者的辨識頻率來改變與所述多個類別中的至少一者對應的所述記憶體的儲存空間的大小。The training network model may be configured to recognize a category corresponding to each of a plurality of pixel blocks included in the input image, and based on the recognition frequency of each of the plurality of categories To change the size of the storage space of the memory corresponding to at least one of the plurality of categories.

所述訓練網路模型可被配置成基於所述辨識頻率而自所述記憶體移除與被辨識少於預定次數的類別對應的紋理片，並將作為所述移除的結果而獲得的儲存空間指配給其他類別。The training network model may be configured to remove texture patches corresponding to categories that have been recognized less than a predetermined number of times from the memory based on the recognition frequency, and store them as a result of the removal Space is assigned to other categories.

所述多個類別可基於平均畫素值、畫素座標、方差、邊緣強度、邊緣方向、或顏色中的至少一者進行分類。The plurality of categories may be classified based on at least one of average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color.

所述至少一個處理器可更被配置成：基於所述紋理片與所述畫素區塊之間的關聯來獲得所述紋理片的加權值，以及藉由將被應用所述加權值的所述紋理片應用至所述畫素區塊來獲得所述輸出影像。The at least one processor may be further configured to: obtain a weighted value of the texture patch based on the association between the texture patch and the pixel block, and obtain the weighted value of the texture patch by applying the weighted value to all The texture sheet is applied to the pixel block to obtain the output image.

所述輸出影像可為4K超高畫質（UHD）影像或8K超高畫質影像。The output image may be a 4K ultra high quality (UHD) image or an 8K ultra high quality image.

根據本揭露的態樣，提供一種影像處理裝置的影像處理方法，所述方法包括：將輸入影像應用至訓練網路模型；以及將與所述輸入影像中所包括的畫素區塊對應的紋理片應用至所述畫素區塊，以獲得輸出影像，其中所述訓練網路模型儲存與基於影像的特性進行分類的多個類別對應的多個紋理片，且基於所述輸入影像來訓練所述多個紋理片中的至少一個紋理片。According to aspects of the present disclosure, there is provided an image processing method of an image processing device, the method comprising: applying an input image to a training network model; and applying a texture corresponding to a pixel block included in the input image Slices are applied to the pixel block to obtain an output image, wherein the training network model stores a plurality of texture slices corresponding to a plurality of categories classified based on the characteristics of the image, and trains all texture slices based on the input image At least one texture sheet among the plurality of texture sheets.

所述訓練網路模型可基於所述畫素區塊的特性來辨識所述多個類別中的類別，獲得與所辨識的所述類別對應的紋理片，將所述畫素區塊與所辨識的所述類別之間的第一相似度和所述紋理片與所辨識的所述類別之間的第二相似度進行比較，並基於所述比較來判斷是否更新所述紋理片。The training network model can identify the categories of the multiple categories based on the characteristics of the pixel block, obtain a texture patch corresponding to the recognized category, and compare the pixel block with the recognized The first similarity between the categories and the second similarity between the texture patch and the recognized category are compared, and based on the comparison, it is determined whether to update the texture patch.

所述訓練網路模型可基於所述比較而以所述畫素區塊來替換與所辨識的所述類別對應的所述紋理片，或者添加所述畫素區塊作為與所辨識的所述類別對應的另一紋理片。The training network model may replace the texture patch corresponding to the recognized category with the pixel block based on the comparison, or add the pixel block as the pixel block corresponding to the recognized Another texture sheet corresponding to the category.

基於根據所述比較得出所述第一相似度小於所述第二相似度，所述訓練網路模型可保持與所辨識的所述類別對應的所述紋理片；以及基於根據所述比較得出所述第一相似度大於所述第二相似度，所述訓練網路模型可基於所述畫素區塊更新所述紋理片。Based on the fact that the first degree of similarity is less than the second degree of similarity based on the comparison, the training network model can maintain the texture patch corresponding to the recognized category; and based on the comparison If the first similarity is greater than the second similarity, the training network model can update the texture patch based on the pixel block.

基於與所辨識的所述類別對應的所述紋理片包括多於一個紋理片，所述訓練網路模型可基於所述畫素區塊與所述多於一個紋理片中的每一紋理片之間的關聯來辨識所述多於一個紋理片中的一者。Based on the fact that the texture patch corresponding to the recognized category includes more than one texture patch, the training network model may be based on the difference between the pixel block and each texture patch in the more than one texture patch. To identify one of the more than one texture patches.

所述訓練網路模型可基於所述至少一個紋理片的儲存時間及所述至少一個紋理片的應用頻率中的至少一者來訓練所述至少一個紋理片。The training network model may train the at least one texture piece based on at least one of the storage time of the at least one texture piece and the application frequency of the at least one texture piece.

基於根據所述畫素區塊的特性而所述畫素區塊不對應於所述多個類別中的一者，所述訓練網路模型可基於所述畫素區塊的所述特性而產生新類別，並可將所述畫素區塊映射及儲存至所述新類別。Based on the characteristics of the pixel block and the pixel block does not correspond to one of the multiple categories, the training network model may be generated based on the characteristics of the pixel block A new category, and the pixel block can be mapped and stored in the new category.

根據本揭露的態樣，提供一種非暫態電腦可讀取記錄媒體，所述非暫態電腦可讀取記錄媒體上面記錄有可由電腦執行的用於實行所述方法的程式。According to an aspect of the present disclosure, a non-transitory computer-readable recording medium is provided, and the non-transitory computer-readable recording medium records a program that can be executed by a computer for implementing the method.

根據本揭露的態樣，提供一種影像處理裝置的影像處理方法，所述方法包括：基於輸入影像來訓練訓練網路模型，所述訓練網路模型儲存與基於影像特性進行分類的多個類別對應的多個紋理片；以及藉由將儲存於所述訓練網路模型中的多個紋理片中與所述輸入影像中所包括的畫素區塊對應的紋理片應用至所述畫素區塊來獲得輸出影像。According to the aspect of the present disclosure, there is provided an image processing method of an image processing device, the method comprising: training a training network model based on an input image, the training network model storing corresponding to multiple categories classified based on image characteristics A plurality of texture patches; and by applying a texture patch corresponding to the pixel block included in the input image among the multiple texture patches stored in the training network model to the pixel block To get the output image.

訓練所述訓練網路模型可包括：基於所述畫素區塊的特性來辨識所述多個類別中的類別；獲得與所辨識的所述類別對應的紋理片；將所述畫素區塊與所辨識的所述類別之間的第一相似度和所述紋理片與所辨識的所述類別之間的第二相似度進行比較；以及基於所述比較來判斷是否更新所述訓練網路模型中的所述紋理片。Training the training network model may include: identifying categories of the multiple categories based on the characteristics of the pixel blocks; obtaining texture patches corresponding to the identified categories; and converting the pixel blocks Compare with the first similarity between the recognized category and the second similarity between the texture patch and the recognized category; and determine whether to update the training network based on the comparison The texture sheet in the model.

所述訓練所述訓練網路模型可更包括基於確定更新所述紋理片而以所述畫素區塊來替換與所辨識的所述類別對應的所述紋理片，或者添加所述畫素區塊作為與所辨識的所述類別對應的另一紋理片。The training of the training network model may further include, based on determining to update the texture patch, replacing the texture patch corresponding to the identified category with the pixel block, or adding the pixel area The block serves as another texture piece corresponding to the recognized category.

所述判斷是否更新所述紋理片可包括：基於根據所述比較得出所述第一相似度小於所述第二相似度，在所述訓練網路模型中保持與所辨識的所述類別對應的所述紋理片；以及基於根據所述比較得出所述第一相似度大於所述第二相似度，基於所述畫素區塊更新所述紋理片。The judging whether to update the texture patch may include: based on the comparison that the first similarity is less than the second similarity, keeping in the training network model corresponding to the identified category The texture patch; and based on the comparison that the first similarity is greater than the second similarity, the texture patch is updated based on the pixel block.

所述獲得與所辨識的所述類別對應的紋理片可包括基於與所辨識的所述類別對應的所述紋理片包括多於一個紋理片，基於所述畫素區塊與所述多於一個紋理片中的每一紋理片之間的關聯來確定所述多於一個紋理片中的一者。The obtaining a texture patch corresponding to the recognized category may include more than one texture patch based on the texture patch corresponding to the recognized category, and based on the pixel block and the more than one texture patch. The association between each texture slice in the texture slice determines one of the more than one texture slices.

所述訓練所述訓練網路模型可包括基於所述多個紋理片中的至少一個紋理片的儲存時間及所述至少一個紋理片的應用頻率中的至少一者來訓練所述訓練網路模型。The training of the training network model may include training the training network model based on at least one of the storage time of at least one texture slice of the plurality of texture slices and the application frequency of the at least one texture slice .

所述訓練所述訓練網路模型可包括基於根據所述畫素區塊的特性而所述畫素區塊不對應於所述多個類別中的一者，在所述訓練網路模型中基於所述畫素區塊的所述特性而產生新類別，並將所述畫素區塊映射及儲存至所述新類別。The training of the training network model may include based on the characteristics of the pixel block and the pixel block does not correspond to one of the multiple categories, in the training network model based on The characteristics of the pixel block generate a new category, and the pixel block is mapped and stored in the new category.

包括技術用語及科學用語在內的本說明書中所使用的所有用語皆具有與熟習先前技術者通常所理解的相同的含義。然而，該些用語可根據熟習此項技術者的意圖、法律或技術闡釋以及新技術的出現而有所變化。另外，一些用語是申請人任意地選擇的。該些用語可被解釋為本文中所定義的含義，且除非另有規定，否則可基於本說明書的全部內容及此項技術中的技術共識來加以解釋。All terms used in this specification, including technical terms and scientific terms, have the same meanings as those commonly understood by those familiar with the prior art. However, these terms may vary according to the intentions of those familiar with the technology, legal or technical interpretation, and the emergence of new technologies. In addition, some terms are arbitrarily selected by the applicant. These terms can be interpreted as meanings defined in this text, and unless otherwise specified, they can be interpreted based on the entire content of this specification and the technical consensus in this technology.

在本說明書中，例如「包括」及「具有（have/has）」等用語應被解釋為指明存在該些特徵（例如，數目、操作、元件或組件），且不排除存在或可能添加其他特徵中的一者或多者。In this specification, terms such as "include" and "have/has" shall be interpreted as indicating the presence of these features (for example, number, operation, elements or components), and do not exclude the presence or possible addition of other features One or more of them.

在本揭露中，「A或B」、「A及B中的至少一者」、「A或B中的至少一者」、「A及/或B中的一者或多者」等表達包括所列項的所有可能的組合。In this disclosure, expressions such as "A or B", "at least one of A and B", "at least one of A or B", "one or more of A and/or B", etc. include All possible combinations of the listed items.

可使用例如「第一」及「第二」等用語來修飾各種元件，而不論次序及/或重要性如何。該些用語僅用於將一個組件與其他組件區分開的目的。Terms such as "first" and "second" can be used to modify various elements, regardless of order and/or importance. These terms are only used for the purpose of distinguishing one component from other components.

當稱一元件（例如，第一構成元件）為「可操作地或通訊地耦合至」或者「連接至」另一元件（例如，第二構成元件）時，應理解，每一構成元件經由另一構成元件（例如，第三構成元件）直接地連接或間接地連接。When an element (for example, a first constituent element) is referred to as being "operably or communicatively coupled to" or "connected to" another element (for example, a second constituent element), it should be understood that each constituent element passes through another A constituent element (for example, a third constituent element) is directly connected or indirectly connected.

單數表達亦包含複數含義，只要所述複數含義在對應的上下文中未傳達不同的含義即可。在本說明書中，例如「包括」及「具有（have/has）」等用語應被解釋為指明說明書中存在該些特徵、數目、操作、元件、組件或其組合，且不排除存在或可能添加其他特徵、數目、操作、元件、組件或其組合中的一者或多者。The singular expression also includes the plural meaning, as long as the plural meaning does not convey different meanings in the corresponding context. In this manual, terms such as "including" and "have/has" shall be interpreted as indicating the presence of these features, numbers, operations, elements, components or combinations thereof, and does not exclude the presence or possible additions One or more of other features, numbers, operations, elements, components, or combinations thereof.

在一個或多個實施例中，「模組」、「單元」或「部分」實行至少一個功能或操作，且可被實現為例如處理器或積體電路等硬體、由處理器執行的軟體或者所述硬體與軟體的組合。另外，多個「模組」、多個「單元」或多個「部分」可被整合至至少一個模組或晶片中且可被實現為至少一個處理器，但應被實現為特定硬體的「模組」、「單元」或「部分」除外。In one or more embodiments, the "module", "unit" or "part" performs at least one function or operation, and can be implemented as hardware such as a processor or an integrated circuit, and software executed by the processor Or a combination of the hardware and software. In addition, multiple "modules", multiple "units" or multiple "parts" can be integrated into at least one module or chip and can be implemented as at least one processor, but should be implemented as specific hardware Except for "module", "unit" or "part".

在本說明書中，用語「使用者」是指使用電子裝置的人或使用電子裝置的裝置（例如，人工智慧電子裝置）。In this manual, the term "user" refers to a person who uses an electronic device or a device that uses an electronic device (for example, an artificial intelligence electronic device).

在下文中，將參照附圖詳細闡述一個或多個實施例。Hereinafter, one or more embodiments will be explained in detail with reference to the accompanying drawings.

圖1是闡釋根據實施例的影像處理裝置100的示例性實施例的視圖。FIG. 1 is a view explaining an exemplary embodiment of an image processing apparatus 100 according to an embodiment.

參照圖1，影像處理裝置100可被實施成電視，但並非僅限於此。影像處理裝置100可被實施成包括顯示功能的任何類型的裝置，例如智慧型電話、平板電腦、膝上型電腦、頭戴式顯示器（head mounted display，HMD）、近眼顯示器（near eye display，NED）、大型顯示器（large format display，LFD）、數位標牌（digital signage）、數位資訊顯示器（digital information display，DID）、視訊牆、投影機顯示器等。1, the image processing device 100 can be implemented as a television, but it is not limited to this. The image processing device 100 can be implemented as any type of device including a display function, such as a smart phone, a tablet computer, a laptop computer, a head mounted display (HMD), a near eye display (NED) ), large format display (LFD), digital signage (digital signage), digital information display (DID), video wall, projector display, etc.

影像處理裝置100可接收各種解析度的影像或各種壓縮影像。舉例而言，影像處理裝置100可接收標準畫質（Standard Definition，SD）影像、高畫質（High Definition，HD）影像、全HD影像、超HD影像（例如，4K·UHD、8K·UHD等）等等。影像處理裝置100可接收壓縮形式（例如，MPEG（例如，MP2、MP4、MP7等）、AVC、H.264、HEVC等）的影像。The image processing device 100 can receive images of various resolutions or various compressed images. For example, the image processing device 100 can receive standard definition (SD) images, high definition (High Definition, HD) images, full HD images, ultra HD images (for example, 4K·UHD, 8K·UHD, etc.) )and many more. The image processing device 100 may receive images in a compressed format (for example, MPEG (for example, MP2, MP4, MP7, etc.), AVC, H.264, HEVC, etc.).

根據實施例，即使影像處理裝置100被實施成UHD TV，亦可輸入SD影像、HD影像、全HD影像等（在下文中稱為低解析度影像），此是由於例如缺少UHD內容。在此種情形中，可使用將輸入低解析度影像10擴展成UHD影像或更高解析度的影像（在下文中稱為高解析度影像）的方法。然而，在先前技術中存在問題：在擴展影像的過程中，影像的紋理模糊且細節劣化。影像的紋理是指被視為影像的相同特徵的區的獨特圖案或形狀。According to the embodiment, even if the image processing device 100 is implemented as a UHD TV, it can also input SD images, HD images, full HD images, etc. (hereinafter referred to as low-resolution images) due to, for example, lack of UHD content. In this case, a method of expanding the input low-resolution video 10 into a UHD video or a higher-resolution video (hereinafter referred to as a high-resolution video) can be used. However, there is a problem in the prior art: in the process of expanding the image, the texture of the image is blurred and the details are degraded. The texture of an image refers to the unique pattern or shape of an area that is regarded as the same feature of the image.

此外，即使輸入高解析度影像，亦可由於影像壓縮而發生紋理丟失，從而可能無法確保細節。隨著畫素的數目增加，數位影像可使用更多的資料，且在壓縮的情形中，因壓縮而引起的紋理丟失是不可避免的。In addition, even if a high-resolution image is input, texture loss may occur due to image compression, and details may not be ensured. As the number of pixels increases, digital images can use more data, and in the case of compression, texture loss due to compression is inevitable.

因此，以下將針對各種情形闡述用於恢復丟失的紋理成分及增強影像的細節的各種實施例。Therefore, various embodiments for recovering the lost texture components and enhancing the details of the image will be described below for various situations.

圖2是闡釋根據實施例的影像處理裝置100的配置的方塊圖。FIG. 2 is a block diagram illustrating the configuration of the image processing apparatus 100 according to the embodiment.

參照圖2，影像處理裝置100可包括記憶體110及處理器120。2, the image processing device 100 may include a memory 110 and a processor 120.

記憶體110可電性連接至處理器120，且可儲存各種實施例中所使用的資料。舉例而言，記憶體110可被實施成內部記憶體，例如唯讀記憶體（read-only memory，ROM）（例如，電性可抹除可程式化唯讀記憶體（electrically erasable programmable read-only memory，EEPROM））、隨機存取記憶體（random access memory，RAM）、或與處理器120分離的記憶體。在此種情形中，端視資料儲存的目的而定，記憶體110可被實施成以下形式：嵌置在影像處理裝置100中的記憶體、或***影像處理裝置100中的可移動記憶體。舉例而言，用於驅動影像處理裝置100的資料可儲存於嵌置在影像處理裝置100中的記憶體中，且用於影像處理裝置100的擴展功能的資料可儲存於可附接至影像處理裝置100或可自影像處理裝置100拆離的記憶體中。嵌置在影像處理裝置100中的記憶體可用以下記憶體中的至少一者來實施：揮發性記憶體（例如，動態RAM（dynamic RAM，DRAM）、靜態RAM（static RAM，SRAM）、同步動態RAM（synchronous dynamic RAM，SDRAM）等）；非揮發性記憶體（例如，一次可程式化ROM（one time programmable ROM，OTPROM）、可程式化ROM（programmable ROM，PROM）、可抹除及可程式化ROM（erasable and programmable ROM，EPROM）、電性可抹除及可程式化ROM（EEPROM）、罩幕ROM、快閃ROM、快閃記憶體（例如，與反及閃存或者反或閃存）、硬驅動機（hard drive）或固態驅動機（solid state drive，SSD））。可自影像處理裝置100移除的記憶體可由以下裝置來實施：記憶卡（例如，緊湊式快閃卡、安全數位（secure digital，SD）卡、微型SD卡、小型SD卡、極限數位（extreme digital，xD）卡等）、可連接至通用串列匯流排（universal serial bus，USB）埠的外部記憶體（例如，USB記憶體）等。The memory 110 can be electrically connected to the processor 120 and can store data used in various embodiments. For example, the memory 110 can be implemented as an internal memory, such as read-only memory (read-only memory, ROM) (for example, electrically erasable programmable read-only memory). memory, EEPROM), random access memory (random access memory, RAM), or a memory separate from the processor 120. In this case, depending on the purpose of data storage, the memory 110 can be implemented in the following forms: a memory embedded in the image processing device 100 or a removable memory inserted into the image processing device 100. For example, the data used to drive the image processing device 100 can be stored in a memory embedded in the image processing device 100, and the data used for the extended functions of the image processing device 100 can be stored in the image processing device that can be attached to the image processing device. The device 100 may be in a memory detachable from the image processing device 100. The memory embedded in the image processing device 100 can be implemented by at least one of the following memory: volatile memory (for example, dynamic RAM (DRAM, DRAM), static RAM (static RAM, SRAM), synchronous dynamic RAM (synchronous dynamic RAM, SDRAM), etc.); non-volatile memory (for example, one time programmable ROM (OTPROM), programmable ROM (programmable ROM, PROM), erasable and programmable ROM (erasable and programmable ROM, EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (for example, reverse and flash memory or reverse or flash memory), Hard drive (hard drive) or solid state drive (solid state drive, SSD)). The memory that can be removed from the image processing device 100 can be implemented by the following devices: memory card (for example, compact flash card, secure digital (SD) card, micro SD card, small SD card, extreme digital (extreme digital, xD) card, etc.), external memory (for example, USB memory) that can be connected to a universal serial bus (USB) port, etc.

記憶體110可儲存用於獲得與輸入影像10中所包括的畫素區塊對應的紋理片的訓練網路模型。訓練網路模型可為基於多個影像的機器學習模型。舉例而言，訓練網路模型可為基於基於多個樣本影像及輸入影像10的卷積神經網路（Convolution Neural Network，CNN）訓練的模型。CNN可為具有為語音處理、影像處理等而設計的特定連接結構的多層神經網路。具體而言，CNN可藉由對畫素進行預處理而以各種方式過濾影像，且辨識影像的特性。舉例而言，CNN可辨識輸入影像10中所包括的預定大小的畫素區塊的特性。訓練網路模型並非僅限於CNN。舉例而言，影像處理裝置100可使用基於各種神經網路（例如，遞歸神經網路（Recurrent Neural Network，RNN）、深度神經網路（Deep Neural Network，DNN）等）的訓練網路模型。The memory 110 can store a training network model for obtaining texture patches corresponding to the pixel blocks included in the input image 10. The training network model can be a machine learning model based on multiple images. For example, the training network model may be a model trained based on a Convolution Neural Network (CNN) based on multiple sample images and input images 10. CNN can be a multilayer neural network with a specific connection structure designed for voice processing, image processing, and so on. Specifically, CNN can filter the image in various ways by preprocessing the pixels, and recognize the characteristics of the image. For example, the CNN can recognize the characteristics of pixel blocks of predetermined size included in the input image 10. Training network models is not limited to CNN. For example, the image processing device 100 may use training network models based on various neural networks (for example, Recurrent Neural Network (RNN), Deep Neural Network (DNN), etc.).

同時，「紋理片」是指應用至畫素區塊以改善畫素區塊的紋理的片。為了方便起見，用語「片」可為考慮到功能而應用的用語，但在實施例中可使用除了用語「片」之外的各種用語。舉例而言，每一片可具有其中多個片值以畫素單元矩陣的形式對齊的結構，且因此可被稱為罩幕。當紋理片被應用至畫素區塊時，畫素區塊的紋理可得到改善，且畫素區塊的細節可得到改善。與不論畫素區塊的特性如何便將紋理片固定至畫素區塊相反，影像處理裝置100可使用訓練網路模型來應用經更新的紋理片。At the same time, "texture film" refers to a film applied to a pixel block to improve the texture of the pixel block. For convenience, the term "片" may be a term applied in consideration of functions, but various terms other than the term "片" may be used in the embodiment. For example, each slice may have a structure in which a plurality of slice values are aligned in the form of a pixel unit matrix, and thus may be called a mask. When the texture patch is applied to the pixel block, the texture of the pixel block can be improved, and the details of the pixel block can be improved. In contrast to fixing the texture patch to the pixel block regardless of the characteristics of the pixel block, the image processing device 100 can use the training network model to apply the updated texture patch.

處理器120可電性連接至記憶體110且控制影像處理裝置100的總體操作。The processor 120 may be electrically connected to the memory 110 and control the overall operation of the image processing device 100.

根據實施例，處理器120可被實施成數位訊號處理器（digital signal processor，DSP）、微處理器或時間控制器（time controller，TCON），但並非僅限於此。處理器120可包括一個或多個中央處理單元（central processing unit，CPU）、微控制器單元（microcontroller unit，MCU）、微處理單元（micro processing unit，MPU）、控制器、應用處理器（application processor，AP）、通訊處理器（communication processor，CP）、高階RISC機器（Advanced RISC Machine，ARM）處理器等，或者可由對應的用語定義。處理器120可被實施成系統晶片（system on chip，SoC）、具有內建處理演算法的大型積體（large scale integration，LSI），或以現場可程式化閘陣列（Field Programmable Gate Array，FPGA）的形式來實施。According to the embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON), but it is not limited to this. The processor 120 may include one or more central processing unit (CPU), microcontroller unit (MCU), micro processing unit (MPU), controller, and application processor (application processor). processor, AP), communication processor (communication processor, CP), advanced RISC machine (Advanced RISC Machine, ARM) processor, etc., or may be defined by corresponding terms. The processor 120 can be implemented as a system on chip (SoC), a large scale integration (LSI) with a built-in processing algorithm, or a Field Programmable Gate Array (FPGA). ) Form to implement.

處理器120可藉由對輸入影像進行處理來獲得輸出影像。處理器120可藉由對輸入影像執行紋理增強處理（texture enhancement process）來獲得輸出影像。輸出影像可為超高畫質（UHD）影像，具體而言，可為4K UHD影像或8K UHD影像，但並非僅限於此。The processor 120 can obtain an output image by processing the input image. The processor 120 may obtain an output image by performing a texture enhancement process on the input image. The output image may be an ultra-high quality (UHD) image, specifically, it may be a 4K UHD image or an 8K UHD image, but it is not limited to this.

根據實施例的處理器120可獲得用於紋理增強處理的紋理片。處理器120可藉由將輸入影像10應用至訓練網路模型來獲得與輸入影像10中所包括的畫素區塊對應的紋理片。畫素區塊是指包括至少一個畫素的一組相鄰畫素。The processor 120 according to an embodiment may obtain a texture patch for texture enhancement processing. The processor 120 can obtain a texture patch corresponding to the pixel block included in the input image 10 by applying the input image 10 to the training network model. A pixel block refers to a group of adjacent pixels including at least one pixel.

圖3是闡釋根據實施例的畫素區塊20的視圖。FIG. 3 is a view explaining the pixel block 20 according to the embodiment.

參照圖3，處理器120可以畫素區塊20為單位來劃分構成輸入影像10的影像訊框的多個畫素且將所述多個畫素輸入至訓練網路模型。根據實施例，處理器120可依序地將構成影像訊框的多個畫素區塊20輸入至訓練網路模型。訓練網路模型可輸出分別與多個畫素區塊20-1、...及20-n對應的紋理片30-1、...、及30-n（參見圖5）。Referring to FIG. 3, the processor 120 may divide a plurality of pixels constituting an image frame of the input image 10 in units of pixel blocks 20 and input the plurality of pixels into the training network model. According to an embodiment, the processor 120 may sequentially input a plurality of pixel blocks 20 constituting the image frame to the training network model. The training network model can output texture patches 30-1,..., and 30-n corresponding to multiple pixel blocks 20-1,..., and 20-n, respectively (see FIG. 5).

處理器120可將輸入影像10劃分或辨識為5×5畫素區塊20，但畫素區塊的大小並非僅限於此。畫素區塊的大小可被實施成N×N（例如3×3、4×4等）、或M×N的各種大小。處理器120可根據輸入影像的解析度（例如，FHD）、輸出影像的解析度（UHD及8K）等中的至少一者而將輸入影像10劃分或辨識成各種大小的畫素區塊20。在下文中，為易於闡釋，其中畫素區塊20以矩陣格式排列在構成輸入影像10的影像訊框中的預定大小的畫素組將被稱為自輸入影像10獲得的畫素區塊20。The processor 120 can divide or recognize the input image 10 into 5×5 pixel blocks 20, but the size of the pixel blocks is not limited to this. The size of the pixel block can be implemented as N×N (for example, 3×3, 4×4, etc.), or various sizes of M×N. The processor 120 can divide or recognize the input image 10 into pixel blocks 20 of various sizes according to at least one of the resolution of the input image (for example, FHD), the resolution of the output image (UHD and 8K), etc. Hereinafter, for ease of explanation, a pixel group of a predetermined size in which the pixel blocks 20 are arranged in a matrix format in an image frame constituting the input image 10 will be referred to as a pixel block 20 obtained from the input image 10.

參照圖2，處理器120可藉由將輸入影像10應用至訓練網路模型來獲得與畫素區塊20對應的紋理片。現將參照圖4對其進行詳細說明。Referring to FIG. 2, the processor 120 can obtain a texture patch corresponding to the pixel block 20 by applying the input image 10 to the training network model. This will now be described in detail with reference to FIG. 4.

圖4是闡釋根據實施例的紋理片30的視圖。FIG. 4 is a view explaining the texture sheet 30 according to the embodiment.

圖4是將構成輸入影像10的畫素中的每一者示出為畫素值的視圖。處理器120可藉由將輸入影像10應用至訓練網路模型來獲得與畫素區塊20對應的紋理片30。所述應用是指將輸入影像10輸入至訓練網路模型，且訓練網路模型的輸出可為紋理片30。FIG. 4 is a view showing each of the pixels constituting the input image 10 as a pixel value. The processor 120 can obtain the texture sheet 30 corresponding to the pixel block 20 by applying the input image 10 to the training network model. The application refers to inputting the input image 10 to the training network model, and the output of the training network model can be a texture slice 30.

訓練網路模型可輸出與輸入影像10中所包括的畫素區塊20對應的紋理片30，且基於畫素區塊20實行訓練。The training network model can output a texture patch 30 corresponding to the pixel block 20 included in the input image 10 and perform training based on the pixel block 20.

訓練網路模型可包括基於影像的各種特性中的任何一者進行分類的多個類別，且包括與所述類別中的每一者對應的紋理片30。舉例而言，訓練網路模型可基於影像的特性中的邊緣方向儲存所分類的所述多個類別，且包括與所述多個類別中的每一者對應的紋理片30。作為另一實例，訓練網路模型可以畫素區塊20為單位而來儲存基於影像的特性中的灰度平均值所分類的多個類別，且包括與所述類別中的每一者對應的紋理片30。The training network model may include multiple categories classified based on any one of various characteristics of the image, and includes texture patches 30 corresponding to each of the categories. For example, the training network model may store the multiple categories classified based on the edge direction in the characteristics of the image, and include texture patches 30 corresponding to each of the multiple categories. As another example, the training network model can store a plurality of categories classified based on the gray-level average value in the characteristics of the image in units of pixel blocks 20, and include the corresponding to each of the categories Texture sheet 30.

影像處理裝置100可包括多個訓練網路模型。影像處理裝置100可包括多個訓練網路模型，例如用於基於邊緣方向劃分類別且對紋理片30實行訓練的第一訓練網路模型、用於基於灰度平均值劃分類別且實行訓練的第二訓練網路模型、用於基於顏色座標劃分類別且實行訓練的第三訓練網路模型等。影像處理裝置100可基於輸入影像10的特性來辨識所述多個訓練網路模型中的任何一者，且將所辨識的訓練網路模型應用至輸入影像10以獲得紋理片30。舉例而言，影像處理裝置100可包括預處理訓練網路模型以基於輸入影像10的特性來辨識所述多個訓練網路模型中的任何一者，從而獲得適合的紋理片30。舉例而言，若構成輸入影像10的所述多個畫素的顏色分佈在相似的顏色範圍內，則預處理訓練網路模型可辨識用於基於邊緣方向劃分類別且基於影像的特性輸出紋理片30的第一訓練網路模型。The image processing device 100 may include multiple training network models. The image processing device 100 may include a plurality of training network models, such as a first training network model for classifying based on the edge direction and training the texture patch 30, and a second training network model for classifying based on the gray average value and training. 2. A training network model, a third training network model for classifying and training based on color coordinates, etc. The image processing device 100 can identify any one of the multiple training network models based on the characteristics of the input image 10 and apply the identified training network model to the input image 10 to obtain a texture patch 30. For example, the image processing device 100 may include a preprocessing training network model to identify any one of the plurality of training network models based on the characteristics of the input image 10 to obtain a suitable texture patch 30. For example, if the colors of the multiple pixels constituting the input image 10 are distributed in a similar color range, the preprocessing training network model can be identified for classifying based on the edge direction and outputting texture patches based on the characteristics of the image 30's first training network model.

訓練網路模型可基於輸入影像10實行訓練。舉例而言，訓練網路模型可辨識輸入影像10中所包括的畫素區塊20對於與所述畫素區塊20對應的類別的第一相似度、以及所獲得的與所述類別相匹配的紋理片30對於所述類別的第二相似度。在此種情形中，例如，若第一相似度大於第二相似度，則訓練網路模型可確定所獲得的紋理片30不適合於輸入影像10的紋理改善，且基於輸入影像10的畫素區塊20實行更新。當輸出構成輸入影像10的各種畫素區塊中的與包括在和畫素區塊20相同的類別中的另一畫素區塊20’對應的紋理片30時，訓練網路模型可輸出基於畫素區塊20更新的紋理片30’，所述經更新的紋理片30’與更新之前的紋理片30相反。因此，自訓練網路模型輸出的紋理片30可適合於輸入影像10的紋理增強。此外，若第二相似度大於第一相似度，則訓練網路模型可確定所獲得的紋理片30適合於輸入影像10的紋理增強且保持紋理片30。The training network model can be trained based on the input image 10. For example, the training network model can recognize the first similarity of the pixel block 20 included in the input image 10 to the category corresponding to the pixel block 20, and the obtained matching with the category The second degree of similarity of the texture sheet 30 to the category. In this case, for example, if the first similarity is greater than the second similarity, the training network model can determine that the obtained texture patch 30 is not suitable for the texture improvement of the input image 10, and is based on the pixel area of the input image 10. Block 20 performs the update. When outputting a texture slice 30 corresponding to another pixel block 20' included in the same category as the pixel block 20 among the various pixel blocks constituting the input image 10, the training network model can output The updated texture piece 30 ′ of the pixel block 20, and the updated texture piece 30 ′ is opposite to the texture piece 30 before the update. Therefore, the texture slice 30 output from the training network model can be suitable for texture enhancement of the input image 10. In addition, if the second degree of similarity is greater than the first degree of similarity, the training network model can determine that the obtained texture patch 30 is suitable for the texture enhancement of the input image 10 and the texture patch 30 is maintained.

訓練網路模型的對多個類別中與畫素區塊20對應的類別進行分類（或辨識）的操作可被稱為分類器、類別辨識器等。若輸入輸入影像10中所包括的畫素區塊20，則分類器可辨識所述多個類別中的適合於畫素區塊20的類別。舉例而言，分類器可辨識畫素區塊20的邊緣方向，且辨識所辨識的邊緣方向與用於定義所述多個類別中的每一者的邊緣方向之間的相似度。分類器可將所述多個類別中的具有最大相似度的類別辨識為與畫素區塊20對應的類別。The operation of training the network model to classify (or identify) the class corresponding to the pixel block 20 among the multiple classes can be referred to as a classifier, a class recognizer, or the like. If the pixel block 20 included in the input image 10 is input, the classifier can identify a category suitable for the pixel block 20 among the multiple categories. For example, the classifier can recognize the edge direction of the pixel block 20, and recognize the similarity between the recognized edge direction and the edge direction used to define each of the plurality of categories. The classifier can identify the category with the greatest similarity among the multiple categories as the category corresponding to the pixel block 20.

訓練網路模型可藉由如下定義：用於對與畫素區塊20對應的類別進行辨識的模型（例如，分類器模型）與用於對畫素區塊20與和畫素區塊20對應的紋理片30的相似度進行比較且對紋理片30實行自主學習的模型的組合。訓練網路模型可為設備上機器學習模型（On-device Machine Learning Model），其中影像處理裝置100在不依賴於外部設備的條件下進行自主訓練。然而，此僅為實例，且應理解，一個或多個其他實施例並非僅限於此。舉例而言，根據另一實施例，訓練網路模型可被實施成使得分類器模型可在設備上（即在影像處理裝置100中）操作，且用於對紋理片實行訓練的模型可基於外部設備或伺服器進行操作。The training network model can be defined as follows: a model used to identify the category corresponding to the pixel block 20 (for example, a classifier model) and a model used to correspond to the pixel block 20 and the pixel block 20 The similarity of the texture pieces 30 is compared, and the combination of the self-learning models is performed on the texture pieces 30. The training network model may be an on-device machine learning model (On-device Machine Learning Model), in which the image processing device 100 performs autonomous training without relying on external equipment. However, this is only an example, and it should be understood that one or more other embodiments are not limited to this. For example, according to another embodiment, the training network model can be implemented so that the classifier model can be operated on the device (ie in the image processing device 100), and the model used to train the texture patch can be based on external Device or server to operate.

因此，訓練網路模型可儲存與基於影像的特性進行分類及訓練的所述多個類別中的每一者對應的紋理片30。在輸出與輸入影像10對應的紋理片時，訓練網路模型可基於輸入影像10中所包括的畫素值來對與所述多個類別中的每一者對應的紋理片30進行訓練。Therefore, the training network model can store texture patches 30 corresponding to each of the multiple categories classified and trained based on the characteristics of the image. When outputting the texture patch corresponding to the input image 10, the training network model can train the texture patch 30 corresponding to each of the plurality of categories based on the pixel values included in the input image 10.

參照圖4，訓練網路模型可基於畫素區塊20的特性來辨識多個類別中的與畫素區塊20對應的單個類別。舉例而言，訓練網路模型可儲存基於影像的各種特性中的邊緣方向（或邊緣圖案）進行分類的多個類別。邊緣是指其中畫素值（或畫素亮度）自低值變為高值，或自高值變為低值的點。因此，邊緣是指根據影像中所包括的各種對象產生的對象之間的邊界。訓練網路模型可辨識所述多個類別中的與畫素區塊20的邊緣方向（或邊界的方向）對應的單個類別。訓練網路模型可辨識所述多個類別中的與畫素區塊20的邊緣方向最相似（或最適合）的訊號類別。訓練網路模型可輸出與所辨識的類別對應的紋理片30。參照圖2，處理器120可藉由將自訓練網路模型輸出的紋理片應用至輸入影像10來實行紋理增強處理。Referring to FIG. 4, the training network model can identify a single category corresponding to the pixel block 20 among multiple categories based on the characteristics of the pixel block 20. For example, the training network model can store multiple categories classified based on the edge direction (or edge pattern) in various characteristics of the image. Edge refers to the point where the pixel value (or pixel brightness) changes from a low value to a high value, or from a high value to a low value. Therefore, the edge refers to the boundary between objects generated from various objects included in the image. The training network model can identify a single category corresponding to the edge direction (or the boundary direction) of the pixel block 20 among the multiple categories. The training network model can identify the signal category that is most similar (or most suitable) to the edge direction of the pixel block 20 among the multiple categories. The training network model can output texture patches 30 corresponding to the recognized category. Referring to FIG. 2, the processor 120 can perform texture enhancement processing by applying texture patches output from the training network model to the input image 10.

圖5是闡釋根據實施例的訓練網路模型的視圖。Fig. 5 is a view explaining a training network model according to an embodiment.

如上所述，訓練網路模型可儲存基於影像的特性進行分類的多個類別、以及與所述多個類別的每一者對應的至少一個紋理片30-1、30-2、…、30-8。參照圖5，訓練網路模型可包括基於影像的特性中的邊緣方向進行分類的第一類別至第n類別。訓練網路模型可包括與所述第一類別至所述第n類別中的每一者對應的紋理片30-1、30-2、…、30-8。影像的特性可包括畫素區塊20中所包括的畫素值的平均值、方差、畫素座標、邊緣強度、邊緣方向、顏色等中的至少一者。根據實施例的訓練網路模型可包括基於畫素值的平均值、方差、畫素座標、邊緣強度、邊緣方向、顏色等中的至少一者進行分類的多個類別。除了上述實例之外，訓練網路模型可基於自畫素區塊20辨識的各種特性產生多個類別且辨識所述多個類別中的哪一類別與畫素區塊20對應。舉例而言，訓練網路模型可基於顏色座標分類出類別，且基於畫素區塊20中所包括的畫素的顏色座標的平均值來辨識與畫素區塊20對應的類別。As described above, the training network model can store multiple categories classified based on the characteristics of the image, and at least one texture piece 30-1, 30-2, ..., 30- corresponding to each of the multiple categories. 8. Referring to FIG. 5, the training network model may include the first category to the nth category classified based on the edge direction in the characteristics of the image. The training network model may include texture patches 30-1, 30-2, ..., 30-8 corresponding to each of the first category to the nth category. The characteristics of the image may include at least one of the average value, variance, pixel coordinates, edge intensity, edge direction, color, etc. of the pixel values included in the pixel block 20. The training network model according to the embodiment may include multiple categories classified based on at least one of the average value of pixel values, variance, pixel coordinates, edge strength, edge direction, color, and the like. In addition to the above examples, the training network model can generate multiple categories based on various characteristics identified from the pixel block 20 and identify which category of the multiple categories corresponds to the pixel block 20. For example, the training network model can classify categories based on color coordinates, and identify the category corresponding to the pixel block 20 based on the average value of the color coordinates of the pixels included in the pixel block 20.

參照圖5，處理器120可以畫素區塊20為單位對構成輸入影像10的影像訊框中所包括的多個畫素進行分類，且將所述多個畫素（即，畫素區塊20-1、...及20-n）輸入至訓練網路模型中。處理器120可將構成影像訊框的所述多個畫素區塊20-1、...及20-n依序地輸入至訓練網路模型中。訓練網路模型可輸出分別與所述多個畫素區塊20-1、...及20-n對應的紋理片30-1、...及30-n。5, the processor 120 may classify the multiple pixels included in the image frame constituting the input image 10 in units of pixel blocks 20, and classify the multiple pixels (ie, pixel blocks) 20-1, ... and 20-n) input to the training network model. The processor 120 may sequentially input the plurality of pixel blocks 20-1, ..., and 20-n constituting the image frame into the training network model. The training network model can output texture patches 30-1,..., and 30-n corresponding to the plurality of pixel blocks 20-1,..., and 20-n, respectively.

舉例而言，訓練網路模型可基於第一畫素區塊20-1的特性來辨識所述多個類別中的與第一畫素區塊20-1對應的類別。在此種情形中，訓練網路模型可基於構成第一畫素區塊20-1的畫素來辨識第一畫素區塊20-1的邊緣方向，且辨識所述多個類別中的哪一類別與所辨識的邊緣方向對應。訓練網路模型可辨識所述多個類別與第一畫素區塊20-1之間的相似度。舉例而言，若第一畫素區塊20-1的邊緣方向是0度，則與第二類別至第八類別（Class #2至Class #8）相比，訓練網路模型可在第一類別（Class #1）中獲得高的相似度（或適應度）。第一類別（Class #1）可指基於0度的邊緣方向定義的類別。訓練網路模型可因此將第一類別（Class #1）辨識為與第一畫素區塊20-1對應的類別。處理器120可藉由訓練網路模型獲得與第一類別（Class #1）對應的第一紋理片30-1。For example, the training network model can identify the category corresponding to the first pixel block 20-1 among the multiple categories based on the characteristics of the first pixel block 20-1. In this case, the training network model can recognize the edge direction of the first pixel block 20-1 based on the pixels constituting the first pixel block 20-1, and recognize which of the multiple categories The category corresponds to the recognized edge direction. The training network model can identify the similarity between the multiple categories and the first pixel block 20-1. For example, if the edge direction of the first pixel block 20-1 is 0 degrees, compared with the second to eighth categories (Class #2 to Class #8), the training network model can be used in the first A high degree of similarity (or fitness) is obtained in the class (Class #1). The first category (Class #1) may refer to a category defined based on an edge direction of 0 degrees. The training network model can therefore recognize the first category (Class #1) as the category corresponding to the first pixel block 20-1. The processor 120 may obtain the first texture piece 30-1 corresponding to the first class (Class #1) by training the network model.

作為另一實例，若第二畫素區塊20-2被辨識成與所述多個類別中的第二類別（Class #2）對應，則訓練網路模型可提供與第二類別（Class #2）對應的第二紋理片30-2。As another example, if the second pixel block 20-2 is identified as corresponding to the second category (Class #2) of the plurality of categories, the training network model can provide the second category (Class #2). 2) The corresponding second texture sheet 30-2.

為了易於闡釋，圖5示出訓練網路模型包括基於邊緣方向的第一類別至第八類別，且所述類別中的每一者包括單個紋理片，即第一紋理片至第八紋理片30-1、...及30-8。然而，應理解，此僅為實例，且一個或多個其他實施例並非僅限於此。For ease of explanation, FIG. 5 shows that the training network model includes the first to eighth categories based on the edge direction, and each of the categories includes a single texture patch, that is, the first texture patch to the eighth texture patch 30 -1,... and 30-8. However, it should be understood that this is only an example, and one or more other embodiments are not limited to this.

同時，若畫素區塊20被辨識成不與基於畫素區塊20的特性的所述多個類別中的任何一者對應，則訓練網路模型可基於畫素區塊20的特性產生新類別，且將畫素區塊20映射及儲存於新類別中。舉例而言，若畫素區塊20與所述多個類別之間的相似度小於臨限值，則訓練網路模型可基於畫素區塊20的特性產生除了所述多個類別之外的新類別。At the same time, if the pixel block 20 is identified as not corresponding to any one of the multiple categories based on the characteristics of the pixel block 20, the training network model can generate a new model based on the characteristics of the pixel block 20 Category, and map and store the pixel block 20 in the new category. For example, if the similarity between the pixel block 20 and the multiple categories is less than the threshold value, the training network model can generate other than the multiple categories based on the characteristics of the pixel block 20 New category.

參照圖5，根據實施例，若第一類別至第八類別與第四畫素區塊20-4之間的相似度是臨限值或更小（或小於臨限值）（即，不存在被辨識為與第四畫素區塊20-4對應的類別），則訓練網路模型可基於第四畫素區塊20-4的特性產生第九類別。舉例而言，若基於邊緣方向對所述多個類別進行分類，則訓練網路模型可辨識構成第四畫素區塊20-4的畫素的邊緣方向且基於所辨識的邊緣方向產生第九類別。訓練網路模型可將第四畫素區塊20-4映射至第九類別且儲存第四畫素區塊20-4。舉例而言，訓練網路模型可儲存第四畫素區塊20-4作為與新產生的第九類別對應的紋理片。5, according to an embodiment, if the similarity between the first to eighth categories and the fourth pixel block 20-4 is a threshold value or less (or less than the threshold value) (ie, there is no If it is identified as a category corresponding to the fourth pixel block 20-4), the training network model can generate a ninth category based on the characteristics of the fourth pixel block 20-4. For example, if the multiple categories are classified based on the edge direction, the training network model can identify the edge direction of the pixels constituting the fourth pixel block 20-4 and generate the ninth pixel based on the identified edge direction. category. The training network model can map the fourth pixel block 20-4 to the ninth category and store the fourth pixel block 20-4. For example, the training network model can store the fourth pixel block 20-4 as a texture patch corresponding to the newly generated ninth category.

若辨識出與對應於畫素區塊20的類別相匹配的紋理片30，則訓練網路模型可基於畫素區塊20與類別之間的相似度以及紋理片30與類別之間的相似度來辨識紋理片30是否已被更新。訓練網路模型可藉由將用於定義類別的參考與畫素區塊20之間的相似度（或適應度）和用於定義類別的參考與和類別相匹配的紋理片30之間的相似度進行比較來辨識是否已實行更新。參照圖5，訓練網路模型可包括如上所述基於邊緣方向進行分類的多個類別。所述多個類別中的第一類別（Class #1）可為邊緣方向被定義成0度的類別，且第五類別（Class #5）可為邊緣方向被定義成90度的類別。若輸入第一畫素區塊20-1，則訓練網路模型可基於第一畫素區塊20-1的邊緣方向來辨識所述多個類別中的具有最大相似度的第一類別（Class #1）。可藉由將第一類別（Class #1）與第一畫素區塊20-1之間的相似度和第一類別（Class #1）與第一紋理片30-1之間的相似度進行比較來辨識第一紋理片30-1是否已被更新。If the texture patch 30 that matches the category corresponding to the pixel block 20 is identified, the training network model can be based on the similarity between the pixel block 20 and the category and the similarity between the texture patch 30 and the category To identify whether the texture sheet 30 has been updated. The training network model can be based on the similarity (or fitness) between the reference used to define the category and the pixel block 20 and the similarity between the reference used to define the category and the texture patch 30 that matches the category. The degree is compared to identify whether the update has been implemented. Referring to FIG. 5, the training network model may include multiple categories classified based on edge directions as described above. The first category (Class #1) of the plurality of categories may be a category where the edge direction is defined as 0 degrees, and the fifth category (Class #5) may be a category where the edge direction is defined as 90 degrees. If the first pixel block 20-1 is input, the training network model can identify the first category with the greatest similarity among the multiple categories based on the edge direction of the first pixel block 20-1 (Class #1). The similarity between the first class (Class #1) and the first pixel block 20-1 and the similarity between the first class (Class #1) and the first texture sheet 30-1 The comparison is used to identify whether the first texture sheet 30-1 has been updated.

現將參照圖6對更新進行詳細說明。The update will now be described in detail with reference to FIG. 6.

圖6是闡釋根據實施例的類別及紋理片30的視圖。FIG. 6 is a view explaining the category and texture sheet 30 according to the embodiment.

參照圖6，訓練網路模型可基於畫素區塊20的特性來辨識所述多個類別中的與畫素區塊20對應的類別。舉例而言，若畫素區塊20包括65度的邊緣方向，則訓練網路模型可自所述第一類別至第八類別（Class #1至Class #8）中辨識由67.5度的邊緣方向定義的第四類別（Class #4）。訓練網路模型可獲得與所辨識的第四類別（Class #4）對應的紋理片30。Referring to FIG. 6, the training network model can identify the category corresponding to the pixel block 20 among the multiple categories based on the characteristics of the pixel block 20. For example, if the pixel block 20 includes an edge direction of 65 degrees, the training network model can identify an edge direction of 67.5 degrees from the first to eighth categories (Class #1 to Class #8) The fourth category defined (Class #4). The training network model can obtain the texture patch 30 corresponding to the recognized fourth category (Class #4).

訓練網路模型可基於畫素區塊20與第四類別（Class #4）之間的相似度以及紋理片30與第四類別（Class #4）之間的相似度來辨識紋理片30是否已被更新。可使用各種類型的相似度量測演算法、適應度量測演算法及機器學習演算法來量測、確定或獲得相似度。舉例而言，可藉由基於灰度值中的至少一者對直方圖進行比較、計算歐幾裡德距離（Euclidean distance）等來辨識相似度的程度。作為另一實例，可附加地或作為另一種選擇基於卷積神經網路（CNN）訓練演算法來辨識相似度的程度。The training network model can identify whether the texture patch 30 is already based on the similarity between the pixel block 20 and the fourth category (Class #4) and the similarity between the texture patch 30 and the fourth category (Class #4). Updated. Various types of similarity measurement algorithms, fitness measurement algorithms, and machine learning algorithms can be used to measure, determine, or obtain similarity. For example, the degree of similarity can be identified by comparing histograms based on at least one of the gray values, calculating Euclidean distance, and so on. As another example, a convolutional neural network (CNN) training algorithm may be additionally or alternatively used to identify the degree of similarity.

舉例而言，當紋理片30的與根據訓練網路模型的另一（例如，先前的）輸入影像、樣本影像等的第四類別（Class #4）相匹配的邊緣方向為50度時，用於定義第四類別（Class #4）的邊緣方向可為67.5度。因此，訓練網路模型可辨識出邊緣方向為65度的畫素區塊20的第一相似度可大於邊緣方向為50度的紋理片30的第二相似度，且畫素區塊20適合於第四類別（Class #4）。訓練網路模型可基於畫素區塊20來替代或更新紋理片30。若輸入輸入影像10中所包括的另一畫素區塊，且所述另一畫素區塊與第四類別（Class #4）對應，則訓練網路模型可輸出基於邊緣方向為65度的畫素區塊20更新的紋理片。處理器120可基於經更新的紋理片產生所述另一畫素區塊的紋理。For example, when the edge direction of the texture patch 30 that matches the fourth category (Class #4) of another (eg, previous) input image, sample image, etc. according to the training network model is 50 degrees, use In defining the fourth category (Class #4), the edge direction can be 67.5 degrees. Therefore, the training network model can recognize that the first similarity of the pixel block 20 with an edge direction of 65 degrees can be greater than the second similarity of the texture sheet 30 with an edge direction of 50 degrees, and the pixel block 20 is suitable for The fourth category (Class #4). The training network model can replace or update the texture patch 30 based on the pixel block 20. If another pixel block included in the input image 10 is input, and the other pixel block corresponds to the fourth category (Class #4), the training network model can output an edge direction of 65 degrees Texture sheet updated in pixel block 20. The processor 120 may generate the texture of the another pixel block based on the updated texture patch.

作為另一實例，當類別和與所述類別相匹配的紋理片30之間的第二相似度大於與畫素區塊對應的類別和畫素區塊20之間的第一相似度時，訓練網路模型可辨識出紋理片30適合於輸入影像10及畫素區塊20的紋理產生，且保持紋理片30不變。As another example, when the second similarity between the category and the texture patch 30 matching the category is greater than the first similarity between the category corresponding to the pixel block and the pixel block 20, the training The network model can recognize that the texture sheet 30 is suitable for texture generation of the input image 10 and the pixel block 20, and keep the texture sheet 30 unchanged.

訓練網路模型可在獲得與輸入影像10中所包括的畫素區塊20對應的紋理片30的過程中更新紋理片30，且產生包括適合於輸入影像10的紋理增強的紋理片30的影像處理模型。The training network model can update the texture patch 30 in the process of obtaining the texture patch 30 corresponding to the pixel block 20 included in the input image 10, and generate an image including the texture patch 30 suitable for the texture enhancement of the input image 10 Processing model.

舉例而言，當訓練網路模型被應用至包括例如森林、草坪等對象的輸入影像10時，訓練網路模型可將構成輸入影像10的畫素區塊20與類別之間的相似度和預先儲存的紋理片30與類別之間的相似度進行比較，以保持預先儲存的紋理片30或者以畫素區塊20來替換（或更新）預先儲存的紋理片30。根據實施例，當訓練網路模型被應用至輸入影像10中所包括的另一畫素區塊時，訓練網路模型可辨識在前面的過程中基於畫素區塊20更新的紋理片。在此種情形中，經更新的紋理片可自輸入影像10獲得，且與同一輸入影像10中所包括的另一個畫素區塊具有高的關聯及高的適應度。因此，處理器120可藉由將經更新的紋理片應用至另一畫素區塊來獲得具有得到改善的紋理及細節的輸出影像。For example, when the training network model is applied to an input image 10 including objects such as forests, lawns, etc., the training network model can calculate the similarity between the pixel blocks 20 and the categories that constitute the input image 10 and the The similarity between the stored texture piece 30 and the category is compared to maintain the pre-stored texture piece 30 or replace (or update) the pre-stored texture piece 30 with the pixel block 20. According to an embodiment, when the training network model is applied to another pixel block included in the input image 10, the training network model can recognize the texture patch updated based on the pixel block 20 in the previous process. In this case, the updated texture patch can be obtained from the input image 10, and has a high correlation and high adaptability with another pixel block included in the same input image 10. Therefore, the processor 120 can obtain an output image with improved texture and detail by applying the updated texture patch to another pixel block.

根據實施例的訓練網路模型可基於與所述多個類別中的每一者對應的紋理片30的儲存時間或應用頻率中的至少一者來訓練紋理片30。The training network model according to the embodiment may train the texture patch 30 based on at least one of the storage time or the application frequency of the texture patch 30 corresponding to each of the plurality of categories.

亦即，根據實施例，訓練網路模型可基於輸入影像10來訓練紋理片30，且亦考慮何時儲存預先儲存的紋理片30。舉例而言，若辨識或確定出與輸入影像10中所包括的畫素區塊20對應的紋理片30的儲存時間經過預定時間（例如，在預定時間段之前），則訓練網路模型可以畫素區塊20來替換紋理片30。當紋理片30的儲存時間長時，與輸入影像10的適應度或合適度以及與匹配關係中的類別的相似度可能低。因此，訓練網路模型可基於輸入影像10中所包括的畫素區塊20實行訓練且更新紋理片30。訓練網路模型可將輸入影像10中所包括的畫素區塊20映射至與畫素區塊20對應的類別的紋理片30，且使用新映射的紋理片30來產生輸入影像10的紋理。That is, according to the embodiment, the training network model can train the texture patch 30 based on the input image 10 and also consider when to store the pre-stored texture patch 30. For example, if it is identified or determined that the storage time of the texture sheet 30 corresponding to the pixel block 20 included in the input image 10 has passed a predetermined time (for example, before a predetermined time period), the training network model can draw The plain block 20 replaces the texture sheet 30. When the storage time of the texture sheet 30 is long, the adaptability or suitability to the input image 10 and the similarity to the category in the matching relationship may be low. Therefore, the training network model can perform training based on the pixel blocks 20 included in the input image 10 and update the texture patch 30. The training network model can map the pixel block 20 included in the input image 10 to a texture patch 30 corresponding to the pixel block 20, and use the newly mapped texture patch 30 to generate the texture of the input image 10.

作為另一實例，若畫素區塊20與類別之間的相似度和紋理片30與類別之間的第二相似度相同，則訓練網路模型可基於紋理片30的儲存時間、應用頻率等來更新紋理片30。舉例而言，當第一相似度與第二相似度相同時，畫素區塊20可能比預先儲存的紋理片30更適合於輸入影像10的紋理產生，且可基於畫素區塊20來更新紋理片30。作為另一種選擇，基於第一相似度與第二相似度相同，訓練網路模型可添加除了紋理片30之外的畫素區塊20。As another example, if the similarity between the pixel block 20 and the category is the same as the second similarity between the texture patch 30 and the category, the training network model can be based on the storage time and application frequency of the texture patch 30, etc. Let's update texture sheet 30. For example, when the first degree of similarity is the same as the second degree of similarity, the pixel block 20 may be more suitable for texture generation of the input image 10 than the pre-stored texture patch 30, and it can be updated based on the pixel block 20 Texture sheet 30. As another option, based on the fact that the first degree of similarity is the same as the second degree of similarity, the training network model can add pixel blocks 20 other than the texture patch 30.

然而，該些僅為實例，且當紋理片30的儲存時間經過預定時間時，一個或多個其他實施例可不更新紋理片30。However, these are only examples, and one or more other embodiments may not update the texture sheet 30 when the storage time of the texture sheet 30 has passed a predetermined time.

此外，根據實施例，訓練網路模型可基於紋理片30的應用頻率來訓練紋理片30。舉例而言，當特定紋理片30被辨識為頻繁用於產生除了當前輸入影像10之外的另一輸入影像（或其他輸入影像，例如，先前的輸入影像）的紋理時，特定紋理片30可與類別具有的高適應度或合適度，且有用地適用於紋理產生。然而，若辨識出特定紋理片30不太頻繁地用於紋理產生，則訓練網路模型可辨識或確定（或者可理解）紋理片30與映射關係中的類別具有較低的適應度或合適度。在此種情形中，訓練網路模型可以輸入影像10中所包括的畫素區塊20來替換紋理片30。In addition, according to an embodiment, the training network model may train the texture patch 30 based on the application frequency of the texture patch 30. For example, when a specific texture patch 30 is recognized as a texture frequently used to generate another input image (or other input images, such as a previous input image) other than the current input image 10, the specific texture patch 30 may It has a high degree of adaptability or suitability to the category, and is usefully suitable for texture generation. However, if it is recognized that the specific texture patch 30 is not frequently used for texture generation, the training network model can recognize or determine (or understand) that the texture patch 30 and the category in the mapping relationship have low fitness or suitability. . In this case, the training network model can input the pixel block 20 included in the image 10 to replace the texture piece 30.

如上所述，在一個或多個實施例中，若基於畫素區塊20的特性將所述多個類別中的特定類別辨識成與畫素區塊20對應的類別，且若與所辨識的類別對應的紋理片30的儲存時間經過預定時間及/或紋理片30的應用頻率小於臨限值（例如，臨限值次數），則訓練網路模型可以畫素區塊20替換紋理片30。As described above, in one or more embodiments, if a specific category of the multiple categories is identified as a category corresponding to the pixel block 20 based on the characteristics of the pixel block 20, and if it is consistent with the identified category The storage time of the texture piece 30 corresponding to the category has passed a predetermined time and/or the application frequency of the texture piece 30 is less than the threshold (for example, the threshold number of times), then the training network model can replace the texture piece 30 with the pixel block 20.

圖7是闡釋根據實施例的用於訓練輸入影像10的模型的視圖。FIG. 7 is a view explaining a model for training the input image 10 according to an embodiment.

參照圖7，訓練網路模型可不儲存與所述多個類別的部分對應的一個或多個紋理片30。舉例而言，訓練網路模型可不儲存分別與第一類別至第八類別對應的第一紋理片至第八紋理片30-1、...、及30-8中的所有者，但是可僅為映射關係所述多個類別中的一些類別儲存紋理片30-1、30-2、…30-8，而所述類別中的其餘類別可不具有儲存及映射至其的對應的紋理片。在此種情形中，訓練網路模型可基於輸入影像10來獲得及儲存紋理片30。舉例而言，當訓練網路模型辨識與輸入影像10中所包括的畫素區塊20對應的類別，且不包括與所辨識的類別對應的紋理片30時，訓練網路模型可將畫素區塊20映射及儲存至所辨識的類別。Referring to FIG. 7, the training network model may not store one or more texture patches 30 corresponding to the portions of the multiple categories. For example, the training network model may not store the owners of the first to eighth texture tiles 30-1,..., and 30-8 corresponding to the first to eighth categories, but only Texture patches 30-1, 30-2,... 30-8 are stored for some of the multiple categories in the mapping relationship, and the remaining categories in the category may not have corresponding texture patches stored and mapped to them. In this case, the training network model can obtain and store texture tiles 30 based on the input image 10. For example, when the training network model recognizes the category corresponding to the pixel block 20 included in the input image 10, and does not include the texture patch 30 corresponding to the recognized category, the training network model can The block 20 is mapped and stored to the recognized category.

同時，以上已闡述了所述類別包括映射到其的僅單個紋理片30，然而應理解，一個或多個其他實施例並非僅限於此。舉例而言，第一類別可包括儲存及映射到其的與第一類別對應的至少兩個紋理片30。根據實施例，訓練網路模型可辨識輸入影像10中所包括的畫素區塊20的類別，且將畫素區塊20添加至所辨識的類別作為紋理片30。在此種情形中，訓練網路模型可不刪除或替換預先儲存的紋理片30，而是可將預先儲存的紋理片30設定為第一紋理片，且將畫素區塊20設定為第二紋理片，且將第一紋理片及第二紋理片映射及儲存至對應的類別。At the same time, it has been explained above that the category includes only a single texture patch 30 mapped to it, but it should be understood that one or more other embodiments are not limited to this. For example, the first category may include at least two texture patches 30 corresponding to the first category stored and mapped to it. According to an embodiment, the training network model can recognize the category of the pixel block 20 included in the input image 10 and add the pixel block 20 to the recognized category as a texture patch 30. In this case, the training network model may not delete or replace the pre-stored texture piece 30, but may set the pre-stored texture piece 30 as the first texture piece, and set the pixel block 20 as the second texture piece. The first texture sheet and the second texture sheet are mapped and stored to the corresponding category.

基於被辨識成與包括多個紋理片30的畫素區塊20對應的紋理片30，訓練網路模型可基於畫素區塊20與所述多個紋理片30中的每一者之間的關聯來辨識所述多個紋理片30中的一者。舉例而言，當與畫素區塊20對應的類別是第四類別，且與第四類別成映射關係的紋理片30包括第一紋理片至第三紋理片30時，訓練網路模型可辨識或確定畫素區塊20與第一紋理片至第三紋理片30中的每一者之間的關聯，且辨識出在所辨識的關聯中具有最大關聯值的紋理片30。具有最大關聯值的紋理片30是指對於畫素區塊20的紋理產生具有最高適應度或合適度的片。訓練網路模型可藉由將所辨識的紋理片30應用至畫素區塊20來產生紋理。Based on the texture patch 30 identified as corresponding to the pixel block 20 including the plurality of texture patches 30, the training network model may be based on the difference between the pixel block 20 and each of the plurality of texture patches 30 One of the plurality of texture patches 30 is identified by association. For example, when the category corresponding to the pixel block 20 is the fourth category, and the texture patch 30 mapped to the fourth category includes the first texture patch to the third texture patch 30, the training network model can recognize Or determine the association between the pixel block 20 and each of the first to third texture tiles 30, and identify the texture tile 30 with the largest association value among the identified associations. The texture slice 30 with the largest correlation value refers to the slice with the highest adaptability or suitability for the texture generation of the pixel block 20. The training network model can generate texture by applying the identified texture patch 30 to the pixel block 20.

圖8是闡釋根據另一實施例的類別的視圖。FIG. 8 is a view explaining categories according to another embodiment.

參照圖8，訓練網路模型可基於影像的一個或多個特性來將畫素區塊20辨識成第一類別至第十六類別中的一者。訓練網路模型可辨識與所分類的類別成映射關係的紋理片30。可將所辨識的紋理片30應用至畫素區塊20。Referring to FIG. 8, the training network model can recognize the pixel block 20 as one of the first to sixteenth categories based on one or more characteristics of the image. The training network model can identify texture patches 30 that have a mapping relationship with the classified category. The identified texture patch 30 can be applied to the pixel block 20.

訓練網路模型可基於各種參考來區分類別。類別的數目可不為固定的或有限的，但是訓練網路模型可刪除多個類別中的特定類別，或者產生除了所述多個類別之外的附加類別。The training network model can distinguish categories based on various references. The number of categories may not be fixed or limited, but the training network model can delete specific categories among multiple categories, or generate additional categories in addition to the multiple categories.

為了易於闡釋，已闡述了基於邊緣方向分類出類別，然而應理解，一個或多個其他實施例並非僅限於此。舉例而言，訓練網路模型可基於顏色座標的分佈分類出第一類別至第n類別，且基於輸入影像10中所包括的畫素區塊20的顏色座標分佈來辨識第一類別至第n類別中的對應類別。對於另一實例，訓練網路模型可基於平均灰度值、灰度值的分佈等分類出第一類別至第n類別。For ease of explanation, the classification based on the edge direction has been described, but it should be understood that one or more other embodiments are not limited to this. For example, the training network model can classify the first to nth categories based on the distribution of color coordinates, and identify the first to nth categories based on the color coordinate distribution of the pixel blocks 20 included in the input image 10 The corresponding category in the category. For another example, the training network model can classify the first to nth categories based on the average gray value, the distribution of gray values, and so on.

圖9是闡釋根據實施例的訓練結果的視圖。Fig. 9 is a view explaining a training result according to the embodiment.

參照圖9，訓練網路模型可提供與構成輸入影像10的所述多個畫素區塊20中的每一者對應的紋理片30，且處理器120可藉由將紋理片30應用至畫素區塊20來獲得具有得到改善的細節的輸出影像。9, the training network model can provide a texture patch 30 corresponding to each of the plurality of pixel blocks 20 constituting the input image 10, and the processor 120 can apply the texture patch 30 to the image The plain blocks 20 are used to obtain output images with improved details.

當訓練網路模型基於輸入影像10中所包括的畫素區塊20實行訓練時，在影像10的輸入之前與之後，訓練網路模型中所包括的所述多個類別及紋理片30可不同。舉例而言，輸入影像之前的訓練網路模型可包括基於先前輸入的另一影像或樣本影像訓練的紋理片30。訓練網路模型可辨識或確定輸入影像10中所包括的畫素區塊20與和畫素區塊20對應的類別之間的相似度，以及與類別映射的紋理片30與所述類別之間的相似度，且基於辨識結果更新紋理片30。舉例而言，訓練網路模型可以畫素區塊20來替換紋理片30，或者保持紋理片30。When the training network model is trained based on the pixel blocks 20 included in the input image 10, the multiple categories and texture patches 30 included in the training network model may be different before and after the input of the image 10 . For example, the training network model before the input image may include a texture patch 30 trained based on another image or sample image previously input. The training network model can identify or determine the similarity between the pixel block 20 included in the input image 10 and the category corresponding to the pixel block 20, and between the texture patch 30 mapped to the category and the category The similarity of, and the texture patch 30 is updated based on the recognition result. For example, the training network model can replace the texture piece 30 with the pixel block 20 or keep the texture piece 30.

參照圖9，與訓練網路模型中所包括的所述多個類別中的部分或一些類別對應的紋理片30可被輸入影像10中所包括的且與那些類別對應的畫素區塊20替換。同時，所述多個類別的中其餘類別可保持映射關係中的紋理片30。9, texture patches 30 corresponding to some or some of the multiple categories included in the training network model can be replaced by pixel blocks 20 included in the input image 10 and corresponding to those categories . At the same time, the remaining categories of the multiple categories can maintain the texture patches 30 in the mapping relationship.

圖5、圖6及圖7示出與由箭頭指示的畫素區塊20對應的類別，且圖9示出根據由箭頭指示的訓練網路模型的訓練結果，紋理片30被畫素區塊20替換。舉例而言，參照圖9，與類別2、類別4及類別6中的每一者對應的紋理片30可被輸入影像10中所包括的畫素區塊20替換。5, 6 and 7 show the category corresponding to the pixel block 20 indicated by the arrow, and FIG. 9 shows that the texture patch 30 is divided into the pixel block according to the training result of the training network model indicated by the arrow 20 replacement. For example, referring to FIG. 9, the texture patch 30 corresponding to each of the category 2, the category 4, and the category 6 may be replaced by the pixel block 20 included in the input image 10.

根據實施例，處理器120可基於紋理片30與畫素區塊20之間的關係來獲得紋理片30的加權值。處理器120可藉由將被應用加權值的紋理片30應用至畫素區塊20來獲得輸出影像。According to an embodiment, the processor 120 may obtain the weighted value of the texture tile 30 based on the relationship between the texture tile 30 and the pixel block 20. The processor 120 can obtain the output image by applying the texture patch 30 to which the weighted value is applied to the pixel block 20.

可將輸入影像10中所包括的畫素區塊20與自訓練網路模型獲得的紋理片30之間的關聯（或相關性）計算出一個值（例如，預定值）。關聯的程度可由被稱為關聯係數的值來表示。舉例而言，相關係數可由-1.0與+1.0之間的值來表示，且不論符號如何，數的絕對值越大，關聯越大。舉例而言，負值可指示負關聯，且正值可指示正關聯。A value (for example, a predetermined value) can be calculated for the association (or correlation) between the pixel block 20 included in the input image 10 and the texture patch 30 obtained from the training network model. The degree of correlation can be represented by a value called the correlation coefficient. For example, the correlation coefficient can be represented by a value between -1.0 and +1.0, and regardless of the sign, the greater the absolute value of the number, the greater the correlation. For example, a negative value may indicate a negative association, and a positive value may indicate a positive association.

舉例而言，值C[n]可獲得為E [I * R [n]] = ii * ri，其中畫素值I = [i0, i1, ..., in-1]包括在畫素區塊20中，且值R[n] = [r0, r1, ..., rn-1]包括在紋理片R[n]中。For example, the value C[n] can be obtained as E [I * R [n]] = ii * ri, where the pixel value I = [i0, i1, ..., in-1] is included in the pixel area In block 20, and the value R[n] = [r0, r1, ..., rn-1] is included in the texture patch R[n].

關聯值可基於以下方程式1獲得，其中目標畫素區塊中所包括的畫素值的平均值是m(I)，且紋理片中所包括的值R[n]的平均值是m(R[n])。The correlation value can be obtained based on the following equation 1, where the average value of the pixel values included in the target pixel block is m(I), and the average value of the value R[n] included in the texture patch is m(R [n]).

[方程式1][Equation 1]

根據另一實施例，紋理片30的平均值可為0。當平均值為0時，儘管應用了紋理片30，但是可保持整個輸入影像10的亮度。根據實施例，當紋理片30的平均值為0時，方程式2基於方程式1表達如下。According to another embodiment, the average value of the texture sheet 30 may be zero. When the average value is 0, although the texture patch 30 is applied, the brightness of the entire input image 10 can be maintained. According to an embodiment, when the average value of the texture sheet 30 is 0, Equation 2 is expressed as follows based on Equation 1.

[方程式2][Equation 2]

基於畫素區塊20與和畫素區塊20對應的紋理片30之間的關聯是臨限值或更大（或大於臨限值），訓練網路模型可保持與畫素區塊20的類別對應的紋理片30。此外，基於畫素區塊20與和畫素區塊20對應的紋理片30之間的關聯是臨限值或更小（或小於臨限值），訓練網路模型可基於畫素區塊20更新紋理片30。處理器120可獲得藉由將所獲得的關聯值乘以預定比例常數獲得的值作為與紋理片30對應的加權值。舉例而言，處理器120可基於關聯值獲得0與1之間的加權值。當根據關聯將加權值0應用至紋理片30時，紋理片30可不被添加至目標畫素區塊20。舉例而言，在平坦區域或銳邊區域中，所有類別與所有紋理片之間的關聯可能為低的，且因此可能不會出現紋理。在此種情形中，可防止邊緣區域中可能出現的振鈴現象（ringing phenomenon），且可防止將不必要的紋理添加至平坦區域。Based on the correlation between the pixel block 20 and the texture sheet 30 corresponding to the pixel block 20 is the threshold value or greater (or greater than the threshold value), the training network model can maintain the same value as the pixel block 20 Texture sheet 30 corresponding to the category. In addition, based on the correlation between the pixel block 20 and the texture sheet 30 corresponding to the pixel block 20 is a threshold value or smaller (or less than the threshold value), the training network model can be based on the pixel block 20 Update texture sheet 30. The processor 120 may obtain a value obtained by multiplying the obtained correlation value by a predetermined proportional constant as a weighted value corresponding to the texture sheet 30. For example, the processor 120 may obtain a weighted value between 0 and 1 based on the correlation value. When the weight value 0 is applied to the texture patch 30 according to the association, the texture patch 30 may not be added to the target pixel block 20. For example, in a flat area or a sharp-edged area, the correlation between all categories and all texture patches may be low, and therefore, no texture may appear. In this case, a ringing phenomenon that may occur in the edge area can be prevented, and unnecessary texture can be prevented from being added to the flat area.

根據另一實施例，可藉由除了上述關聯之外的各種成本函數來獲得畫素區塊20與紋理片30之間的相似度資訊。舉例而言，可將均方誤差（Mean Square Error，MSE）、絕對差之和（Sum of Absolute Difference，SAD）、中位數絕對偏差（Median Absolute Deviation，MAD）及關聯用作確定相似度的成本函數。舉例而言，當應用MSE時，可計算目標畫素區塊的均方誤差且可自MSE視點獲得目標畫素區塊20與紋理片30之間的相似度。在此種情形中，可基於MSE差來確定相似度權重。According to another embodiment, the similarity information between the pixel block 20 and the texture sheet 30 can be obtained by various cost functions other than the aforementioned correlation. For example, mean square error (MSE), sum of absolute difference (Sum of Absolute Difference, SAD), Median Absolute Deviation (MAD) and correlation can be used to determine similarity Cost function. For example, when MSE is applied, the mean square error of the target pixel block can be calculated and the similarity between the target pixel block 20 and the texture sheet 30 can be obtained from the MSE viewpoint. In this case, the similarity weight can be determined based on the MSE difference.

處理器120可將所獲得的權重分別應用至紋理片30，且藉由將被應用權重的紋理片30應用至畫素區塊20來獲得輸出影像。應用可指使與被應用權重的紋理片對應的區域中所包括的值加目標畫素區塊20中所包括的每一畫素區塊值的方法。然而，應理解，一個或多個其他實施例並非僅限於此，且可實行除了添加之外的附加處理或其他處理。The processor 120 may apply the obtained weights to the texture tiles 30 respectively, and obtain the output image by applying the texture tiles 30 to which the weights are applied to the pixel block 20. The application may refer to a method of adding the value included in the region corresponding to the texture patch to which the weight is applied to the value of each pixel block included in the target pixel block 20. However, it should be understood that one or more other embodiments are not limited to this, and additional processing other than addition or other processing may be performed.

根據另一實施例，當獲得紋理片30時，處理器120可對紋理片30應用頻率濾波，或者將被應用頻率濾波的紋理片30應用至目標畫素區塊。處理器120可在將紋理片30添加至輸入影像之前應用頻率濾波，且改變紋理片30的頻率範圍。舉例而言，處理器120可藉由使用高通濾波器產生高頻紋理，或者使用低通濾波器產生低頻紋理。方程式3表示藉由使經濾波的紋理(Filter (T))加輸入影像I來獲得輸出影像(O)的過程。According to another embodiment, when the texture tile 30 is obtained, the processor 120 may apply frequency filtering to the texture tile 30, or apply the texture tile 30 to which the frequency filter is applied to the target pixel block. The processor 120 may apply frequency filtering before adding the texture tile 30 to the input image, and change the frequency range of the texture tile 30. For example, the processor 120 may generate a high-frequency texture by using a high-pass filter, or use a low-pass filter to generate a low-frequency texture. Equation 3 represents the process of obtaining the output image (O) by adding the filtered texture (Filter (T)) to the input image I.

[方程式3][Equation 3]

舉例而言，處理器120可對紋理片30應用低通濾波器，例如高斯模糊（Gaussian blurring）（或高斯濾波）。高斯模糊可為使用基於高斯可能性分佈的高斯濾波器進行模糊的方法，且若將高斯濾波器應用至紋理片30，則可阻擋高頻分量（high-frequency component）且可實行模糊。處理器120可對紋理片30中所包括的所有畫素值實行高斯濾波，且獲得模糊紋理片30’。處理器120可藉由將模糊紋理片30’應用至畫素區塊20來獲得輸出影像。For example, the processor 120 may apply a low-pass filter, such as Gaussian blurring (or Gaussian filter), to the texture tile 30. The Gaussian blur may be a method of blurring using a Gaussian filter based on the Gaussian probability distribution, and if the Gaussian filter is applied to the texture sheet 30, high-frequency components can be blocked and blurring can be performed. The processor 120 may perform Gaussian filtering on all the pixel values included in the texture tile 30, and obtain the blurred texture tile 30'. The processor 120 can obtain the output image by applying the blurred texture sheet 30' to the pixel block 20.

同時，上述影像處理（即紋理增強處理）可在影像縮放之前或之後實行。舉例而言，可在縮放之後實行影像處理以將低解析度影像放大至高解析度影像，或者可在對輸入影像進行解碼的過程中在實行影像處理之後實行縮放。At the same time, the aforementioned image processing (ie, texture enhancement processing) can be performed before or after image scaling. For example, image processing can be performed after scaling to enlarge a low-resolution image to a high-resolution image, or scaling can be performed after image processing is performed in the process of decoding the input image.

根據另一實施例的訓練網路模型可獲得與類別對應且被應用不同加權值的多個紋理片。The training network model according to another embodiment can obtain multiple texture patches corresponding to the category and applied with different weighting values.

舉例而言，訓練網路模型可辨識與畫素區塊20對應的類別，且獲得與所述類別對應的第一紋理片至第n紋理片。訓練網路模型可辨識畫素區塊20與第一紋理片至第n紋理片中的每一者之間的關聯。舉例而言，訓練網路模型可基於畫素區塊20與第一紋理片之間的關聯來獲得第一加權值，且基於畫素區塊20與第二紋理片之間的關聯來獲得第二加權值。訓練網路模型可將第一加權值乘以第一紋理片，且將第二加權值乘以第二紋理片，且將第一加權值所乘的第一紋理片及第二加權值所乘的第二紋理片應用至畫素區塊20，從而獲得輸出影像。For example, the training network model can identify the category corresponding to the pixel block 20, and obtain the first to nth texture patches corresponding to the category. The training network model can identify the association between the pixel block 20 and each of the first to nth texture slices. For example, the training network model can obtain the first weight value based on the association between the pixel block 20 and the first texture piece, and obtain the first weight value based on the association between the pixel block 20 and the second texture piece. Two weighted value. The training network model can multiply the first weight value by the first texture piece, and multiply the second weight value by the second texture piece, and multiply the first texture piece multiplied by the first weight value and the second weight value The second texture sheet of is applied to the pixel block 20 to obtain an output image.

根據實施例，可根據關聯將加權值確定在預定範圍中（例如，在0與1之間）。舉例而言，當畫素區塊20與所獲得的紋理片30之間的關聯最小時，訓練網路模型可將加權值確定為0，當關聯最大時，可將加權值確定為1，且可確定使得關聯可在最小值與最大值之間線性增加的加權值。According to an embodiment, the weighting value may be determined in a predetermined range (for example, between 0 and 1) according to the association. For example, when the correlation between the pixel block 20 and the obtained texture piece 30 is the smallest, the training network model can determine the weighting value as 0, and when the correlation is the largest, the weighting value can be determined as 1, and It is possible to determine a weighting value that allows the association to increase linearly between the minimum and maximum values.

圖10是闡釋根據另一實施例的類別的視圖。Fig. 10 is a view explaining categories according to another embodiment.

參照圖10，訓練網路模型可在實行訓練的過程中為每一類別添加或移除紋理片30。10, the training network model can add or remove texture patches 30 for each category during the training process.

根據實施例，訓練網路模型可移除（例如，自記憶體實體地移除或邏輯地移除）特定類別中所包括的紋理，或者在特定類別中儲存多個紋理片以基於輸入影像10中所包括的多個畫素區塊來實行訓練。因此，訓練網路模型可為多個類別的每一者指配相同的儲存空間來儲存紋理片，或者為特定類別指配與其他類別的儲存空間相比更大的儲存空間。According to embodiments, the training network model can remove (for example, physically or logically remove from memory) textures included in a specific category, or store multiple texture patches in a specific category based on the input image 10 Multiple pixel blocks included in the training. Therefore, the training network model can allocate the same storage space for each of multiple categories to store texture patches, or allocate a larger storage space for a specific category than other types of storage space.

根據實施例，訓練網路模型可辨識輸入影像10中所包括的所述多個畫素區塊中的每一者的類別，且基於所述多個類別中的每一者的辨識頻率來改變與所述多個類別中的至少一者對應的記憶體110的儲存空間的大小。舉例而言，訓練網路模型可將用於儲存紋理片的附加儲存空間指配給根據辨識頻率而被辨識多於預定頻率的類別以增加記憶體110的儲存空間的大小。預定頻率，作為實例，可為20%，指示特定類別被辨識為多於畫素區塊的總數目的20%。然而，應理解，此僅為實例，且一個或多個其他實施例並非僅限於此。舉例而言，預定頻率可根據一個或多個其他實施例而變化，例如10%、15%、30%、50%等。作為另一實例，訓練網路模型可基於辨識頻率而增加與最頻繁辨識的類別（或者預定數目的最頻繁辨識的類別，例如最頻繁辨識的類別、第二最頻繁辨識的類別及第三最頻繁辨識的類別）對應的儲存空間的大小。According to an embodiment, the training network model can recognize the category of each of the plurality of pixel blocks included in the input image 10, and change based on the recognition frequency of each of the plurality of categories The size of the storage space of the memory 110 corresponding to at least one of the plurality of categories. For example, the training network model can assign additional storage space for storing texture patches to categories that are recognized more than a predetermined frequency according to the recognition frequency to increase the storage space of the memory 110. The predetermined frequency, as an example, may be 20%, indicating that the specific category is recognized as more than 20% of the total number of pixel blocks. However, it should be understood that this is only an example, and one or more other embodiments are not limited to this. For example, the predetermined frequency may vary according to one or more other embodiments, such as 10%, 15%, 30%, 50%, etc. As another example, the training network model can be increased based on the frequency of recognition and the most frequently recognized categories (or a predetermined number of the most frequently recognized categories, such as the most frequently recognized category, the second most frequently recognized category, and the third most frequently recognized category). The size of the storage space corresponding to the frequently identified category).

舉例而言，基於輸入影像10中所包括的所述多個畫素區塊中的與第四類別對應的多個畫素區塊，訓練網路模型可增加記憶體10上的與第四類別對應的儲存空間的大小。For example, based on a plurality of pixel blocks corresponding to the fourth category among the plurality of pixel blocks included in the input image 10, training the network model can increase the memory 10 and the fourth category The size of the corresponding storage space.

根據實施例，基於被辨識為與第四類別對應的畫素區塊，訓練網路模型可辨識畫素區塊與第四類別之間的第一相似度，以及預先儲存在第四類別中的紋理片與第四類別之間的第二相似度。在此種情形中，訓練網路模型可基於第一相似度小於第二相似度而保持預先儲存的紋理片，且可另外地將畫素區塊儲存於第四類別中。在此種情形中，預先儲存的紋理片可在畫素區塊之前（或優先於畫素區塊）。According to an embodiment, based on the pixel block identified as corresponding to the fourth category, the training network model can recognize the first similarity between the pixel block and the fourth category, and pre-stored in the fourth category The second degree of similarity between the texture patch and the fourth category. In this case, the training network model can maintain the pre-stored texture patches based on the first similarity being less than the second similarity, and can additionally store the pixel blocks in the fourth category. In this case, the pre-stored texture sheet can be before the pixel block (or prior to the pixel block).

作為另一實例，訓練網路模型可基於第一相似度大於第二相似度而另外地將畫素區塊儲存於第四類別中。預先儲存的紋理片的優先級可被改變至較低的位置，且畫素區塊可具有較預先儲存的紋理片高的優先級。As another example, the training network model may additionally store the pixel block in the fourth category based on the first similarity being greater than the second similarity. The priority of the pre-stored texture slice can be changed to a lower position, and the pixel block can have a higher priority than the pre-stored texture slice.

作為又一實例，訓練網路模型可改變記憶體110的儲存空間的大小，使得可基於所述多個類別中的每一者的辨識頻率而將預定數目的紋理片儲存於最頻繁辨識的類別中，且可將小於預定數目的另一預定數目的紋理片儲存於第二最頻繁類別中。舉例而言，訓練網路模型可改變儲存空間的大小，使得最多可將10個紋理片儲存於最頻繁辨識的第四類別中，且最多可將6個紋理片儲存於第二最頻繁辨識的第二類別中。特定數目僅為實例，且應理解，可儲存的紋理片的數目可變化。As another example, the training network model can change the size of the storage space of the memory 110, so that a predetermined number of texture patches can be stored in the most frequently identified category based on the recognition frequency of each of the multiple categories , And another predetermined number of texture patches smaller than the predetermined number can be stored in the second most frequent category. For example, the training network model can change the size of the storage space so that up to 10 texture patches can be stored in the fourth most frequently identified category, and up to 6 texture patches can be stored in the second most frequently identified category In the second category. The specific number is only an example, and it should be understood that the number of texture patches that can be stored may vary.

應理解，訓練網路模型可不總是添加畫素區塊作為與所辨識的類別對應的紋理片，且若畫素區塊與所辨識的類別之間的相似度小於預定值，則可不添加畫素區塊。舉例而言，若畫素區塊與所辨識的類別之間的相似度小於50%，則訓練網路模型可不添加畫素區塊作為所辨識的類別的紋理片。It should be understood that the training network model may not always add pixel blocks as texture patches corresponding to the recognized category, and if the similarity between the pixel block and the recognized category is less than a predetermined value, no painting may be added. Plain block. For example, if the similarity between the pixel block and the recognized category is less than 50%, the training network model may not add the pixel block as a texture patch of the recognized category.

根據實施例，在辨識輸入影像10中所包括的所述多個畫素區塊中的每一者的類別時，訓練網路模型可自記憶體110移除與被辨識少於預定次數（或少於預定頻率）的類別對應的紋理片。訓練網路模型可將記憶體110的儲存空間重新指配給一個或多個其他類別。According to an embodiment, when recognizing the type of each of the plurality of pixel blocks included in the input image 10, the training network model can be removed from the memory 110 and be recognized less than a predetermined number of times (or Less than the predetermined frequency) corresponding to the texture patch. The training network model can reallocate the storage space of the memory 110 to one or more other categories.

舉例而言，作為辨識所述多個畫素區塊中的每一者的類別的結果，當與第三類別對應的畫素區塊的數目小於預定數目時，訓練網路模型可移除預先儲存於第三類別中的一個或多個紋理片，且將用於儲存紋理片的儲存空間指配於其他類別中。因此，訓練網路模型可增加另一類別的儲存空間的大小，進而使得所述多個紋理片可被儲存於最頻繁辨識的類別中。For example, as a result of identifying the category of each of the plurality of pixel blocks, when the number of pixel blocks corresponding to the third category is less than a predetermined number, the training network model can remove the pre- One or more texture tiles stored in the third category, and the storage space for storing the texture tiles is allocated to other categories. Therefore, training the network model can increase the size of the storage space of another category, so that the multiple texture patches can be stored in the most frequently recognized category.

作為另一實例，訓練網路模型可基於辨識頻率移除最不頻繁辨識的類別，且將預先指配給該類別的儲存空間重新指配給一個或多個其他類別。As another example, the training network model can remove the least frequently recognized category based on the recognition frequency, and re-assign the storage space assigned to the category in advance to one or more other categories.

圖11是闡釋根據實施例的影像處理裝置100’的詳細配置的方塊圖。Fig. 11 is a block diagram explaining the detailed configuration of the image processing apparatus 100' according to the embodiment.

參照圖11，影像處理裝置100’可包括記憶體110、處理器120、輸入器130、顯示器140、輸出器150及使用者介面160。以下可省略對圖2的配置的冗餘說明。11, the image processing device 100' may include a memory 110, a processor 120, an input 130, a display 140, an output 150, and a user interface 160. The redundant description of the configuration of FIG. 2 may be omitted below.

根據實施例，記憶體110可被實施成用於儲存各種操作中所產生的資料的單個記憶體。According to an embodiment, the memory 110 may be implemented as a single memory for storing data generated in various operations.

根據另一實施例，記憶體110可被實施成包括多個記憶體，例如第一記憶體至第三記憶體。According to another embodiment, the memory 110 may be implemented to include a plurality of memories, such as a first memory to a third memory.

第一記憶體可儲存藉由輸入器130輸入的影像（例如，影像訊框）的至少部分。第一記憶體可儲存輸入影像訊框的至少局部區域。所述至少局部區域可為實行影像處理所必需或使用的區域。根據實施例，第一記憶體可被實施成N行記憶體。舉例而言，N行記憶體可為在垂直方向上具有17行容量的記憶體，但本揭露並非僅限於此。在此種情形中，當輸入1080畫素（解析度為1,920×1,080）的全HD影像時，可於第一記憶體中僅儲存全HD影像的17行中的影像區域。由於第一記憶體的記憶體容量可能因硬體限制而受到限制，因此可儲存輸入影像訊框的局部區域以用於影像處理。第二記憶體可為用於儲存至少一個所獲得的紋理片30的記憶體，且根據各種實施例實施成各種大小的記憶體。舉例而言，根據實施例，當記憶體被實施成獲得及儲存待應用至輸入影像10的與輸入影像10的各個畫素值對應的所有紋理成分時，第二記憶體可被實施成等於或大於輸入影像10的大小。根據另一實施例，在以與第一記憶體的大小對應的影像單位應用紋理成分的情形中，或者在以畫素行為基礎應用以畫素行為單位獲得的紋理成分的情形中，記憶體可被實施成適合於影像處理的大小。第二記憶體是指指配給記憶體110的整個區域的訓練網路模型的記憶體區域。The first memory can store at least part of an image (for example, an image frame) input through the input 130. The first memory can store at least a partial area of the input image frame. The at least partial area may be an area necessary or used for performing image processing. According to an embodiment, the first memory may be implemented as N rows of memory. For example, the N-row memory may be a memory with a capacity of 17 rows in the vertical direction, but the disclosure is not limited to this. In this case, when a full HD video with 1080 pixels (resolution of 1,920×1,080) is input, only the image area in the 17 rows of the full HD video can be stored in the first memory. Since the memory capacity of the first memory may be limited due to hardware limitations, a local area of the input image frame can be stored for image processing. The second memory may be a memory for storing at least one obtained texture sheet 30, and may be implemented as a memory of various sizes according to various embodiments. For example, according to an embodiment, when the memory is implemented to obtain and store all texture components corresponding to the pixel values of the input image 10 to be applied to the input image 10, the second memory can be implemented to be equal to or Larger than the size of the input image 10. According to another embodiment, in the case of applying the texture component in the image unit corresponding to the size of the first memory, or in the case of applying the texture component obtained in the unit of pixel behavior on the basis of pixel behavior, the memory may It is implemented to a size suitable for image processing. The second memory refers to the memory area of the training network model assigned to the entire area of the memory 110.

第三記憶體可為用於儲存輸出影像的記憶體，所述輸出影像是藉由應用所獲得的紋理成分而被處理且被實施成根據各種實施例的各種大小的記憶體的影像。舉例而言，當第三記憶體被實施成藉由應用與輸入影像10的畫素值對應的紋理成分來獲得及顯示輸出影像時，第三記憶體可被實施成等於或大於輸入影像10的大小的大小。根據另一實施例，當第三記憶體以與第一記憶體的大小對應的影像位單位或者以與片大小對應的行單位輸出影像時，第三記憶體可被實施成適合於儲存影像的大小。The third memory may be a memory for storing output images, which are images processed by applying texture components obtained and implemented into memories of various sizes according to various embodiments. For example, when the third memory is implemented to obtain and display the output image by applying texture components corresponding to the pixel values of the input image 10, the third memory can be implemented to be equal to or larger than the input image 10. The size of the size. According to another embodiment, when the third memory outputs the image in image bit units corresponding to the size of the first memory or in line units corresponding to the slice size, the third memory can be implemented as suitable for storing images. size.

然而，當輸出影像在第一記憶體或第二記憶體中被重寫時，或者當輸出影像在未被儲存的條件下被直接顯示或輸出（例如，傳輸或提供至外部顯示設備）時，可能不需要或使用第三記憶體。However, when the output image is overwritten in the first memory or the second memory, or when the output image is directly displayed or output (for example, transmitted or provided to an external display device) without being stored, The third memory may not be needed or used.

輸入器130可接收各種類型的內容，例如影像訊號。舉例而言，輸入器140可藉由例如以下通信方法自外部伺服器（例如，源設備）、外部儲存媒體（例如，USB）、外部伺服器（例如，網路或雲端儲存）等以流式方法或下載方法接收影像訊號：基於存取點的Wi-Fi（AP-based Wi-Fi）（WiFi、無線局部區域網路）、藍芽、紫峰（Zigbee）、有線/無線局部區域網路（wired/wireless Local Area Network，LAN）、廣域網（Wide Area Network，WAN）、乙太網、IEEE-1394、高畫質多媒體介面（High Definition Multimedia Interface，HDMI）、行動高畫質鏈路（Mobile High-Definition Link，MHL）、通用串列匯流排（USB）、顯示埠（Display Port，DP）、霹靂、視訊圖形陣列（Video Graphic Array，VGA）埠、紅綠藍（RGB）埠、D-超小型（D-subminiature，D-SUB）、數位可視介面（Digital Visual Interface，DVI）等。影像訊號可為數位訊號，但本揭露並非僅限於此。The input 130 can receive various types of content, such as video signals. For example, the input device 140 can stream from an external server (for example, a source device), an external storage medium (for example, USB), an external server (for example, network or cloud storage), etc. through the following communication methods. Method or download method to receive video signal: AP-based Wi-Fi (WiFi, wireless local area network), Bluetooth, Zigbee, wired/wireless local area network ( wired/wireless Local Area Network (LAN), Wide Area Network (WAN), Ethernet, IEEE-1394, High Definition Multimedia Interface (HDMI), Mobile High Definition Link (Mobile High -Definition Link, MHL), Universal Serial Bus (USB), Display Port (DP), Thunderbolt, Video Graphic Array (VGA) port, red-green-blue (RGB) port, D-Ultra Small size (D-subminiature, D-SUB), digital visual interface (Digital Visual Interface, DVI), etc. The image signal may be a digital signal, but this disclosure is not limited to this.

顯示器140可被實施成例如以下等各種形式：液晶顯示器（liquid crystal display，LCD）、有機發光二極體（organic light-emitting diode，OLED）、發光二極體（light-emitting diode，LED）、微型LED、矽上液晶（liquid crystal on silicon，LCoS）、數位光處理（Digital Light Processing，DLP）、量子點（quantum dot，QD）顯示面板等。The display 140 may be implemented in various forms such as the following: liquid crystal display (LCD), organic light-emitting diode (OLED), light-emitting diode (LED), Micro LED, liquid crystal on silicon (LCoS), digital light processing (DLP), quantum dot (QD) display panels, etc.

輸出器150可輸出聲音訊號。The output unit 150 can output audio signals.

舉例而言，輸出器150可將由處理器120處理的數位聲音訊號轉換成類比聲音訊號，且放大並輸出類比聲音訊號。在此種情形中，輸出器150可包括至少一個揚聲器單元、數位至類比（digital-to-analog，D/A）轉換器、音訊放大器等，輸出器150輸出至少一個通道。舉例而言，輸出器150可包括分別再現左通道及右通道的左通道揚聲器及/或右通道揚聲器。然而，本揭露並非僅限於此。輸出器150可被實施成各種形式。對於另一個實例，輸出器150可被實施成再現左通道、右通道及中間通道的聲條的形式。For example, the output unit 150 can convert the digital audio signal processed by the processor 120 into an analog audio signal, and amplify and output the analog audio signal. In this case, the output unit 150 may include at least one speaker unit, a digital-to-analog (D/A) converter, an audio amplifier, etc., and the output unit 150 outputs at least one channel. For example, the output unit 150 may include a left channel speaker and/or a right channel speaker that respectively reproduce the left channel and the right channel. However, this disclosure is not limited to this. The output 150 may be implemented in various forms. For another example, the output 150 may be implemented in the form of reproducing sound bars of the left channel, the right channel, and the middle channel.

使用者介面160可被實施成按鈕、觸控板、軌跡板（trackpad）、可旋轉撥號盤、滑鼠、鍵盤等中的至少一者，及/或被實施成觸控螢幕、能夠實行上述顯示功能及操作輸入功能的遙控接收單元等。按鈕可包括形成於影像處理裝置100’的主體外部的一個或多個區域（例如前部、側部、後部等）中的各種類型的輸入構件，例如機械按鈕、觸控板、撥號盤等。The user interface 160 can be implemented as at least one of a button, a touchpad, a trackpad, a rotatable dial, a mouse, a keyboard, etc., and/or can be implemented as a touch screen, capable of performing the above display Remote control receiving unit for functions and operation input functions, etc. The buttons may include various types of input members, such as mechanical buttons, touch pads, dials, etc., formed in one or more areas (e.g., front, side, rear, etc.) outside the main body of the image processing device 100'.

此外，可另外應用用於在影像處理之前去除輸入影像的噪聲的濾波。舉例而言，可藉由應用根據預定的指導過濾輸入影像的平滑濾波器（例如高斯濾波器、引導濾波器等）來去除不同的噪聲。In addition, a filter for removing noise of the input image before image processing can be additionally applied. For example, different noises can be removed by applying a smoothing filter (such as a Gaussian filter, a guided filter, etc.) that filters the input image according to a predetermined guideline.

圖12是闡釋根據實施例的用於訓練及使用訓練網路模型的影像處理裝置的處理器1200的配置的方塊圖。FIG. 12 is a block diagram illustrating the configuration of a processor 1200 of an image processing apparatus for training and using a trained network model according to an embodiment.

參照圖12，處理器1200可包括訓練單元1210（例如，訓練器）及辨識單元1220（例如，辨識器）中的至少一者。圖11的處理器120可與影像處理裝置100的處理器1200或資料訓練伺服器對應。12, the processor 1200 may include at least one of a training unit 1210 (for example, a trainer) and a recognition unit 1220 (for example, a recognizer). The processor 120 of FIG. 11 may correspond to the processor 1200 of the image processing device 100 or the data training server.

訓練單元1210可產生或訓練具有用於辨識畫素區塊20的類別的參考的辨識模型，以及具有用於根據類別獲得與畫素區塊20對應的紋理片30的參考的辨識模型。訓練單元1210可使用收集的訓練資料產生具有確定參考的辨識模型。The training unit 1210 may generate or train a recognition model with a reference for recognizing the category of the pixel block 20 and a recognition model with a reference for obtaining the texture patch 30 corresponding to the pixel block 20 according to the category. The training unit 1210 can use the collected training data to generate an identification model with a certain reference.

舉例而言，訓練單元1210可藉由使用影像中所包括的畫素區塊20作為訓練資料來產生、訓練或重新開始（例如，更新）用於確定與畫素區塊20對應的類別的辨識模型。For example, the training unit 1210 can generate, train, or restart (for example, update) by using the pixel block 20 included in the image as training data to determine the identification of the category corresponding to the pixel block 20 model.

作為另一實例，訓練單元1210可將畫素區塊20與類別之間的相似度和紋理片30與類別之間的相似度進行比較，且產生、訓練或重新開始辨識模型以用於判斷紋理片30是否已被更新。As another example, the training unit 1210 may compare the similarity between the pixel block 20 and the category and the similarity between the texture patch 30 and the category, and generate, train, or restart the recognition model for determining the texture. Whether slice 30 has been updated.

辨識單元1220可使用預定資料或預定類型的資料（例如，輸入影像10）作為訓練辨識模式的輸入資料，且估測預定資料中所包括的辨識目標或情況。The recognition unit 1220 can use predetermined data or predetermined types of data (for example, the input image 10) as input data for the training recognition mode, and estimate the recognition targets or conditions included in the predetermined data.

舉例而言，辨識單元1220可使用輸入影像10的畫素區塊20作為訓練的辨識模型的輸入資料且辨識畫素區塊20的類別及紋理片30。For example, the recognition unit 1220 can use the pixel block 20 of the input image 10 as the input data of the trained recognition model and recognize the type of the pixel block 20 and the texture sheet 30.

訓練單元1210的至少部分及辨識單元1220的至少部分可被實施成軟體模組及/或被製造成一個或多個硬體晶片形式以安裝於電子裝置（例如影像處理裝置100）上。舉例而言，可將訓練單元1210及辨識單元1220中的至少一者製造成硬體晶片形式以僅用於人工智慧（AI），或者製造成待安裝於各種類型的電子裝置上的現有的通用處理器（例如，CPU或應用處理器）或圖形處理器（例如，GPU）的一部分。用於（例如專用於）人工智慧（AI）的硬體晶片可為專用於概率計算的處理器，所述專用於概率計算的處理器具有較先前技術中的通用處理器高的並行處理效能，從而在例如機器訓練等人工智慧領域中快速實行算術運算。當訓練單元1210及辨識單元1220被實施成軟體模組（或者包含指令的程式模組）時，軟體模組可為或者可儲存於非暫態電腦可讀取媒體（transitory computer readable media）中。在此種情形中，可由作業系統（operating system，OS）、預定應用及/或一個或多個指令來提供軟體模組。作為另外一種選擇，軟體模組中的一些軟體模組可由作業系統（OS）來提供，且軟體模組中的一些軟體模組可由預定應用提供。At least part of the training unit 1210 and at least part of the identification unit 1220 can be implemented as a software module and/or manufactured in the form of one or more hardware chips to be installed on an electronic device (such as the image processing device 100). For example, at least one of the training unit 1210 and the identification unit 1220 can be manufactured in the form of a hardware chip to be used only for artificial intelligence (AI), or manufactured as an existing general-purpose device to be installed on various types of electronic devices. A part of a processor (for example, CPU or application processor) or graphics processor (for example, GPU). The hardware chip used for (for example, dedicated to) artificial intelligence (AI) may be a processor dedicated to probability calculation, and the processor dedicated to probability calculation has higher parallel processing performance than general-purpose processors in the prior art, Thus, arithmetic operations can be quickly implemented in artificial intelligence fields such as machine training. When the training unit 1210 and the identification unit 1220 are implemented as software modules (or program modules containing instructions), the software modules can be or can be stored in a non-transitory computer readable media (transitory computer readable media). In this case, the software module can be provided by an operating system (OS), a predetermined application, and/or one or more commands. Alternatively, some software modules in the software module can be provided by an operating system (OS), and some software modules in the software module can be provided by a predetermined application.

在此種情形中，可將訓練單元1210及辨識單元1220安裝於單個影像處理裝置100上，或者單獨安裝於每一或多個處理裝置（例如，多個影像處理裝置）上。舉例而言，訓練單元1210及辨識單元1220中的一者可被包括於影像處理裝置100中，且另一者可被包括於外部伺服器中。另外，可將由訓練單元1210建立的模型資訊以有線方式或無線方式提供至辨識單元1220，且可將輸入至訓練單元1210中的資料提供至訓練單元1210作為附加訓練資料。In this case, the training unit 1210 and the recognition unit 1220 can be installed on a single image processing device 100, or separately installed on each or multiple processing devices (for example, multiple image processing devices). For example, one of the training unit 1210 and the recognition unit 1220 may be included in the image processing device 100, and the other may be included in an external server. In addition, the model information created by the training unit 1210 can be provided to the identification unit 1220 in a wired or wireless manner, and the data input into the training unit 1210 can be provided to the training unit 1210 as additional training data.

圖13是闡釋根據實施例的影像處理方法的流程圖。FIG. 13 is a flowchart illustrating an image processing method according to an embodiment.

根據圖13的影像處理方法，在操作S1310處，可藉由將輸入影像應用至訓練網路模型來獲得與輸入影像中所包括的畫素區塊對應的紋理片。According to the image processing method of FIG. 13, at operation S1310, the texture patch corresponding to the pixel block included in the input image can be obtained by applying the input image to the training network model.

在操作S1320處，可藉由將所獲得的紋理片應用至畫素區塊來獲得輸出影像。In operation S1320, an output image may be obtained by applying the obtained texture patch to the pixel block.

訓練網路模型可儲存與基於所述影像的一個或多個特性進行分類的多個類別中的每一者對應的紋理片，且基於所述輸入影像來訓練與所述多個類別中的每一者對應的所述紋理片。The training network model may store texture patches corresponding to each of a plurality of categories classified based on one or more characteristics of the image, and train and each of the plurality of categories based on the input image One corresponding to the texture sheet.

訓練網路模型可基於畫素區塊的特性來辨識所述多個類別中的一者，輸出與所辨識的所述類別對應的紋理片，且將所述畫素區塊與所辨識的所述類別之間的第一相似度和所述紋理片與所辨識的所述類別之間的第二相似度進行比較以辨識是否更新所述紋理片。The training network model can recognize one of the multiple categories based on the characteristics of the pixel block, output a texture patch corresponding to the recognized category, and compare the pixel block with all the recognized categories The first similarity between the categories is compared with the second similarity between the texture patch and the recognized category to identify whether to update the texture patch.

訓練網路模型可基於第一相似度及第二相似度而以所述畫素區塊來替換與所辨識的類別對應的紋理片，或者添加所述畫素區塊作為與所辨識的所述類別對應的紋理片。The training network model can replace the texture patch corresponding to the recognized category with the pixel block based on the first similarity and the second similarity, or add the pixel block as the recognized The texture sheet corresponding to the category.

若基於所述比較結果得出所述第一相似度小於所述第二相似度，則所述訓練網路模型可保持與所辨識的所述類別對應的所述紋理片。此外，若所述第一相似度大於所述第二相似度，則所述訓練網路模型可基於所述畫素區塊更新所述紋理片。If the first degree of similarity is less than the second degree of similarity based on the comparison result, the training network model may maintain the texture patch corresponding to the recognized category. In addition, if the first degree of similarity is greater than the second degree of similarity, the training network model may update the texture patch based on the pixel block.

當（例如，基於）與所辨識的所述類別對應的所述紋理片包括多個紋理片時，所述訓練網路模型可基於所述畫素區塊與所述多個紋理片中的每一紋理片之間的關聯來辨識所述多個紋理片中的一者。When (for example, based on) the texture patch corresponding to the recognized category includes multiple texture patches, the training network model may be based on the pixel block and each of the multiple texture patches. The association between a texture patch is used to identify one of the plurality of texture patches.

所述訓練網路模型可基於與所述類別中的每一者對應的紋理片的儲存時間及所述紋理片的應用頻率中的至少一者來訓練所述紋理片。The training network model may train the texture patch based on at least one of the storage time of the texture patch corresponding to each of the categories and the application frequency of the texture patch.

此外，當（例如，基於）基於所述畫素區塊的所述特性而確定出所述畫素區塊不對應於所述多個類別中的一者時，所述訓練網路模型可基於所述畫素區塊的所述特性而產生新類別，將所述畫素區塊映射至所述新類別。In addition, when it is determined (for example, based on) that the pixel block does not correspond to one of the multiple categories based on the characteristic of the pixel block, the training network model may be based on The characteristic of the pixel block generates a new category, and the pixel block is mapped to the new category.

所述多個類別可為基於畫素值的平均值、方差、畫素座標、邊緣強度、邊緣方向、或顏色中的至少一者進行分類。The multiple categories may be classified based on at least one of the average value, variance, pixel coordinates, edge intensity, edge direction, or color of pixel values.

操作S1320處的獲得輸出影像可包括基於所獲得的紋理片與畫素區塊之間的關聯來獲得紋理片的加權值，且藉由將被應用所述加權值的所述紋理片應用至所述畫素區塊來輸出所述輸出影像。Obtaining the output image at operation S1320 may include obtaining a weighted value of the texture patch based on the obtained association between the texture patch and the pixel block, and by applying the texture patch to which the weighted value is applied to all The pixel block is used to output the output image.

所述輸出影像可為4K超高畫質（UHD）影像或8K超高畫質影像，但應理解，一個或多個其他實施例並非僅限於此。事實上，應理解，本概念可被應用至任何解析度的影像（包括小於4K的影像及大於8K的影像）的放大或輸出。The output image may be a 4K ultra high quality (UHD) image or an 8K ultra high quality image, but it should be understood that one or more other embodiments are not limited to this. In fact, it should be understood that the concept can be applied to the enlargement or output of images with any resolution (including images less than 4K and images greater than 8K).

可將各種實施例應用至所有類型的電子裝置，包括影像接收裝置（例如機上盒、及音訊/視訊接收器、媒體流設備等）、或任何類型的影像處理裝置。Various embodiments can be applied to all types of electronic devices, including image receiving devices (such as set-top boxes, audio/video receivers, media streaming equipment, etc.), or any type of image processing device.

可以能夠使用軟體、硬體或其組合的由電腦或類似設備讀取的記錄媒體來實施上述各種實施例。在一些情形中，可由處理器120、1200本身來實施本文中所闡述的實施例。根據軟體實施方案，可以單獨的軟體模組來實施例如本文中所述的程序及功能等的實施例。所述軟體模組中的每一者可實行本文中所述的功能及操作中的一者或多者。The various embodiments described above can be implemented using a recording medium that can be read by a computer or the like, which is software, hardware or a combination thereof. In some cases, the embodiments described herein may be implemented by the processors 120, 1200 itself. According to the software implementation solution, a separate software module can be used to implement embodiments such as the procedures and functions described herein. Each of the software modules can perform one or more of the functions and operations described herein.

可將用於實行根據上述各種實施例的設備的處理操作的電腦指令儲存於非暫態電腦可讀取媒體中。當由特定設備的處理器執行時，儲存於非暫態電腦可讀取媒體中的電腦指令使得特定設備對根據上述各種實施例的設備實行處理操作。The computer instructions used to implement the processing operations of the devices according to the various embodiments described above can be stored in a non-transitory computer readable medium. When executed by the processor of a specific device, computer instructions stored in a non-transitory computer readable medium cause the specific device to perform processing operations on the device according to the various embodiments described above.

所述非暫態電腦可讀取媒體是指半永久地儲存資料而非在極短的時間內儲存資料的媒體（例如，暫存器、高速緩衝記憶體及記憶體），且可由裝置讀取。具體而言，上述各種應用或程式可儲存於例如以下非暫態電腦可讀取媒體中：光碟（compact disc，CD）、數位多功能磁碟（digital versatile disk，DVD）、硬碟、藍光碟、通用串列匯流排（USB）記憶條、記憶卡及唯讀記憶體（ROM），且可提供上述各種應用或程式。The non-transitory computer-readable medium refers to a medium that stores data semi-permanently instead of storing data in a very short period of time (for example, a register, a cache memory, and a memory), and can be read by a device. Specifically, the above-mentioned various applications or programs can be stored in non-transitory computer readable media such as: compact disc (CD), digital versatile disk (DVD), hard disk, Blu-ray disc , Universal serial bus (USB) memory stick, memory card and read-only memory (ROM), and can provide the above-mentioned various applications or programs.

儘管已示出且闡述了一些實施例，但熟習此項技術者應明白可在不背離本揭露的原理及精神的條件下對該些實施例做出改變。因此，本發明概念的範圍不應被解釋為受以上實施例限制，而是由至少隨附申請專利範圍及其等效內容界定。Although some embodiments have been shown and described, those skilled in the art should understand that changes can be made to these embodiments without departing from the principle and spirit of the present disclosure. Therefore, the scope of the concept of the present invention should not be construed as being limited by the above embodiments, but defined by at least the scope of the attached patent application and its equivalent content.

10:輸入影像/輸入低解析度影像 20:畫素區塊 20-1:第一畫素區塊/畫素區塊 20-2:第二畫素區塊/畫素區塊 20-3:畫素區塊 30:紋理片 30-1:第一紋理片/紋理片 30-2:第二紋理片/紋理片 30-8:第八紋理片/紋理片 100、100’:影像處理裝置 110:記憶體 120、1200:處理器 130:輸入器 140:顯示器 150:輸出器 160:使用者介面 1210:訓練單元 1220:辨識單元 S1310、S1320:操作 Class #1:第一類別 Class #2:第二類別 Class #3:第三類別 Class #4:第四類別 Class #5:第五類別 Class #6:第六類別 Class #7:第七類別 Class #8:第八類別10: Input image/input low-resolution image 20: pixel block 20-1: The first pixel block/pixel block 20-2: Second pixel block/pixel block 20-3: Pixel block 30: texture sheet 30-1: The first texture sheet/texture sheet 30-2: Second texture sheet/texture sheet 30-8: Eighth texture sheet/texture sheet 100, 100’: Image processing device 110: memory 120, 1200: processor 130: Input 140: display 150: output device 160: User Interface 1210: Training Unit 1220: Identification Unit S1310, S1320: Operation Class #1: The first category Class #2: The second category Class #3: The third category Class #4: Fourth category Class #5: The fifth category Class #6: The sixth category Class #7: seventh category Class #8: Eighth category

結合附圖閱讀以下說明，本揭露的特定實施例的以上及其他態樣、特徵及優點將更顯而易見，在附圖中：圖1是闡釋根據實施例的影像處理裝置的示例性實施例的視圖。圖2是闡釋根據實施例的影像處理裝置的配置的方塊圖。圖3是闡釋根據實施例的畫素區塊的視圖。圖4是闡釋根據實施例的紋理片的視圖。圖5是闡釋根據實施例的訓練網路模型的視圖。圖6是闡釋根據實施例的類別及紋理片的視圖。圖7是闡釋根據實施例的用於訓練輸入影像的模型的視圖。圖8是闡釋根據另一實施例的類別的視圖。圖9是闡釋根據實施例的訓練結果的視圖。圖10是闡釋根據另一實施例的類別的視圖。圖11是示出根據實施例的影像處理裝置的詳細配置的方塊圖。圖12是闡釋根據實施例的用於訓練及使用訓練網路模型的處理器的配置的方塊圖。圖13是闡釋根據實施例的影像處理方法的流程圖。Reading the following description in conjunction with the accompanying drawings, the above and other aspects, features and advantages of the specific embodiments of the present disclosure will be more apparent. In the accompanying drawings: Fig. 1 is a view explaining an exemplary embodiment of an image processing apparatus according to an embodiment. FIG. 2 is a block diagram explaining the configuration of the image processing apparatus according to the embodiment. FIG. 3 is a view explaining a pixel block according to an embodiment. Fig. 4 is a view explaining a texture sheet according to the embodiment. Fig. 5 is a view explaining a training network model according to an embodiment. Fig. 6 is a view explaining categories and texture patches according to the embodiment. Fig. 7 is a view explaining a model for training an input image according to an embodiment. FIG. 8 is a view explaining categories according to another embodiment. Fig. 9 is a view explaining a training result according to the embodiment. Fig. 10 is a view explaining categories according to another embodiment. FIG. 11 is a block diagram showing the detailed configuration of the image processing apparatus according to the embodiment. FIG. 12 is a block diagram illustrating the configuration of a processor for training and using a training network model according to an embodiment. FIG. 13 is a flowchart illustrating an image processing method according to an embodiment.

100:影像處理裝置 100: Image processing device

110:記憶體 110: memory

120:處理器 120: processor

Claims

一種影像處理裝置，包括：記憶體，被配置成儲存至少一個指令；以及處理器，電性連接至所述記憶體，其中所述處理器藉由執行所述至少一個指令而被配置成：將輸入影像應用至訓練網路模型，以及將與所述輸入影像中所包括的畫素區塊對應的紋理片應用至所述畫素區塊，以獲得輸出影像，其中所述訓練網路模型儲存與基於影像的特性進行分類的多個類別對應的多個紋理片，且被配置成基於所述輸入影像來訓練所述多個紋理片中的至少一個紋理片。An image processing device, including: The memory is configured to store at least one command; and A processor, electrically connected to the memory, The processor is configured by executing the at least one instruction to: Apply the input image to the training network model, and Applying a texture patch corresponding to the pixel block included in the input image to the pixel block to obtain an output image, The training network model stores multiple texture patches corresponding to multiple categories classified based on the characteristics of the image, and is configured to train at least one texture patch of the multiple texture patches based on the input image.

如請求項1所述的裝置，其中所述訓練網路模型被配置成基於所述畫素區塊的特性來辨識所述多個類別中的類別，獲得與所辨識的所述類別對應的紋理片，將所述畫素區塊與所辨識的所述類別之間的第一相似度和所述紋理片與所辨識的所述類別之間的第二相似度進行比較，並基於所述比較來判斷是否更新所述紋理片。The device according to claim 1, wherein the training network model is configured to recognize the categories of the plurality of categories based on the characteristics of the pixel block, and obtain a texture corresponding to the recognized category Compare the first similarity between the pixel block and the recognized category and the second similarity between the texture patch and the recognized category, and based on the comparison To determine whether to update the texture sheet.

如請求項2所述的裝置，其中所述訓練網路模型被配置成基於所述比較而以所述畫素區塊來替換與所辨識的所述類別對應的所述紋理片，或者添加所述畫素區塊作為與所辨識的所述類別對應的另一紋理片。The device according to claim 2, wherein the training network model is configured to replace the texture patch corresponding to the recognized category with the pixel block based on the comparison, or add all The pixel block is used as another texture patch corresponding to the recognized category.

如請求項2所述的裝置，其中：基於根據所述比較得出所述第一相似度小於所述第二相似度，所述訓練網路模型被配置成保持與所辨識的所述類別對應的所述紋理片；以及基於根據所述比較得出所述第一相似度大於所述第二相似度，所述訓練網路模型被配置成基於所述畫素區塊更新所述紋理片。The device according to claim 2, wherein: Based on the fact that the first degree of similarity is less than the second degree of similarity based on the comparison, the training network model is configured to maintain the texture patch corresponding to the recognized category; and Based on finding that the first similarity is greater than the second similarity according to the comparison, the training network model is configured to update the texture patch based on the pixel block.

如請求項2所述的裝置，其中，基於與所辨識的所述類別對應的所述紋理片包括多於一個紋理片，所述訓練網路模型被配置成基於所述畫素區塊與所述多於一個紋理片中的每一紋理片之間的關聯來辨識所述多於一個紋理片中的一者。The device according to claim 2, wherein the texture patch corresponding to the recognized category includes more than one texture patch, and the training network model is configured to be based on the pixel block and the The association between each texture piece in more than one texture piece is described to identify one of the more than one texture pieces.

如請求項1所述的裝置，其中所述訓練網路模型被配置成基於所述至少一個紋理片的儲存時間及所述至少一個紋理片的應用頻率中的至少一者來訓練所述至少一個紋理片。The apparatus according to claim 1, wherein the training network model is configured to train the at least one texture piece based on at least one of the storage time of the at least one texture piece and the application frequency of the at least one texture piece Texture tablets.

如請求項1所述的裝置，其中，基於根據所述畫素區塊的特性而確定出所述畫素區塊不對應於所述多個類別中的一者，所述訓練網路模型被配置成基於所述畫素區塊的所述特性而產生新類別，並將所述畫素區塊映射及儲存至所述新類別。The apparatus according to claim 1, wherein the training network model is determined by determining that the pixel block does not correspond to one of the multiple categories based on the characteristics of the pixel block It is configured to generate a new category based on the characteristics of the pixel block, and map and store the pixel block to the new category.

如請求項1所述的裝置，其中所述訓練網路模型被配置成辨識與所述輸入影像中所包括的多個畫素區塊中的每一者對應的類別，並基於所述多個類別中的每一者的辨識頻率來改變與所述多個類別中的至少一者對應的所述記憶體的儲存空間的大小。The device according to claim 1, wherein the training network model is configured to recognize a category corresponding to each of a plurality of pixel blocks included in the input image, and based on the plurality of pixel blocks The recognition frequency of each of the categories changes the size of the storage space of the memory corresponding to at least one of the multiple categories.

如請求項8所述的裝置，其中所述訓練網路模型被配置成基於所述辨識頻率而自所述記憶體移除與被辨識少於預定次數的類別對應的紋理片，並將作為所述移除的結果而將所獲得的儲存空間指配給其他類別。The device according to claim 8, wherein the training network model is configured to remove from the memory texture patches corresponding to categories recognized less than a predetermined number of times based on the recognition frequency, and use them as the According to the removal results, the obtained storage space is assigned to other categories.

如請求項1所述的裝置，其中所述多個類別是基於平均畫素值、畫素座標、方差、邊緣強度、邊緣方向、或顏色中的至少一者進行分類。The device according to claim 1, wherein the multiple categories are classified based on at least one of average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color.

如請求項1所述的裝置，其中所述處理器更被配置成：基於所述紋理片與所述畫素區塊之間的關聯來獲得所述紋理片的加權值，以及藉由將被應用所述加權值的所述紋理片應用至所述畫素區塊來獲得所述輸出影像。The apparatus according to claim 1, wherein the processor is further configured to: Obtaining the weighted value of the texture patch based on the association between the texture patch and the pixel block, and The output image is obtained by applying the texture patch to which the weighting value is applied to the pixel block.

如請求項1所述的裝置，其中所述輸出影像是4K超高畫質（UHD）影像或8K超高畫質影像。The device according to claim 1, wherein the output image is a 4K ultra high quality (UHD) image or an 8K ultra high quality image.

一種影像處理裝置的影像處理方法，所述方法包括：將輸入影像應用至訓練網路模型；以及將與所述輸入影像中所包括的畫素區塊對應的紋理片應用至所述畫素區塊，以獲得輸出影像，其中所述訓練網路模型儲存與基於影像的特性進行分類的多個類別對應的多個紋理片，且基於所述輸入影像來訓練所述多個紋理片中的至少一個紋理片。An image processing method of an image processing device, the method comprising: Apply the input image to the training network model; and Applying a texture patch corresponding to the pixel block included in the input image to the pixel block to obtain an output image, The training network model stores multiple texture patches corresponding to multiple categories classified based on the characteristics of the image, and trains at least one texture patch of the multiple texture patches based on the input image.

如請求項11所述的方法，其中所述訓練網路模型基於所述畫素區塊的特性來辨識所述多個類別中的類別，獲得與所辨識的所述類別對應的紋理片，將所述畫素區塊與所辨識的所述類別之間的第一相似度和所述紋理片與所辨識的所述類別之間的第二相似度進行比較，並基於所述比較來判斷是否更新所述紋理片。The method according to claim 11, wherein the training network model recognizes the categories of the plurality of categories based on the characteristics of the pixel blocks, obtains texture patches corresponding to the recognized categories, and The first similarity between the pixel block and the identified category is compared with the second similarity between the texture patch and the identified category, and based on the comparison, it is determined whether Update the texture sheet.

如請求項14所述的方法，其中所述訓練網路模型基於所述比較而以所述畫素區塊來替換與所辨識的所述類別對應的所述紋理片，或者添加所述畫素區塊作為與所辨識的所述類別對應的另一紋理片。The method according to claim 14, wherein the training network model replaces the texture patch corresponding to the recognized category with the pixel block based on the comparison, or adds the pixel The block serves as another texture piece corresponding to the recognized category.