WO2024051298A1

WO2024051298A1 - Image processing method, apparatus, and computing device

Info

Publication number: WO2024051298A1
Application number: PCT/CN2023/103166
Authority: WO
Inventors: 陈畅; 彭竞阳; 宋风龙; 李家丞; 熊志伟
Original assignee: 华为技术有限公司; 中国科学技术大学
Priority date: 2022-09-09
Filing date: 2023-06-28
Publication date: 2024-03-14
Also published as: CN115578272A

Abstract

An image processing method, comprising: acquiring a first image; for any pixel point in the first image, determining an image block corresponding to any pixel point by using a plurality of lookup tables associated with any pixel point, the image block comprising a single pixel point or a plurality of pixel points, the index area of each lookup table comprising any pixel point, and the index areas of different lookup tables being different and at least partially the same; according to the image block corresponding to each pixel point in the first image, obtaining a second image, the resolution of the second image being higher than or equal to the resolution of the first image; and outputting the second image. Therefore, during the image processing process, for each pixel point, a plurality of lookup tables are used for cooperative work, so that the receptive field is expanded while maintaining linear growth of computing amount and storage amount, thereby improving the image processing performance.

Description

一种图像处理方法、装置及计算设备An image processing method, device and computing equipment

本申请要求于2022年9月9日提交中国国家知识产权局、申请号为202211102553.9、申请名称为“一种图像处理方法、装置及计算设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 9, 2022, with application number 202211102553.9 and the application title "An image processing method, device and computing device", the entire content of which is incorporated by reference. incorporated in this application.

技术领域Technical field

本申请涉及人工智能(artificial intelligence，AI)技术领域，尤其涉及一种图像处理方法、装置及计算设备。This application relates to the technical field of artificial intelligence (AI), and in particular to an image processing method, device and computing equipment.

背景技术Background technique

图像处理是底层计算机视觉的主要研究方向。近年来，基于卷积神经网络(convolutional neural network，CNN)的方法在图像处理性能方面取得了令人瞩目的效果。一般越复杂的神经网络可以取得越优的效果。然而受到运行速度和功耗的限制，在手机等计算能力较弱的终端设备上进行图像处理的网络一般不能过于复杂。尽管可以通过精心设计的网络结构或专用计算引擎(比如图形处理器)来减轻这种约束，但是相应的硬件成本和功耗仍然增加了现有CNN方法在终端设备上部署的难度。另一方面，在智能车等终端设备上，高速行驶中变道与突发险情处理等应用场景对图像处理方法的运行时延有较高要求。因此，如何设计一种高效的图像处理方法，满足部署在终端设备时低功耗与低时延的要求，同时具有较好的图像处理性能，是目前亟需解决的技术问题。Image processing is the main research direction of underlying computer vision. In recent years, methods based on convolutional neural networks (CNN) have achieved impressive results in image processing performance. Generally, more complex neural networks can achieve better results. However, due to limitations of operating speed and power consumption, the network for image processing on terminal devices with weak computing capabilities such as mobile phones generally cannot be too complex. Although this constraint can be alleviated through a carefully designed network structure or a dedicated computing engine (such as a graphics processor), the corresponding hardware cost and power consumption still increase the difficulty of deploying existing CNN methods on terminal devices. On the other hand, on terminal devices such as smart cars, application scenarios such as lane changing during high-speed driving and handling emergencies have higher requirements on the running delay of image processing methods. Therefore, how to design an efficient image processing method that meets the requirements of low power consumption and low latency when deployed in terminal equipment, while having good image processing performance, is an urgent technical problem that needs to be solved.

发明内容Contents of the invention

本申请提供了一种图像处理方法、装置、计算设备、计算设备集群、计算机存储介质及计算机产品，能够满足部署在终端设备时低功耗与低时延的要求，同时具有较好的图像处理性能。This application provides an image processing method, device, computing equipment, computing equipment cluster, computer storage medium and computer product, which can meet the requirements of low power consumption and low delay when deployed in terminal equipment, and at the same time have better image processing performance.

第一方面，本申请提供一种图像处理方法，该方法包括：获取第一图像；对于第一图像中的任意一个像素点，利用N个(N≥2)与该任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块，图像块中包括单个或多个像素点，其中，每个查找表的索引区域均包含该任意一个像素点，且不同的查找表的索引区域不同且至少有部分相同；根据第一图像中各个像素点所对应的图像块，得到第二图像，其中，第二图像的分辨率高于或等于第一图像的分辨率；输出第二图像。In a first aspect, this application provides an image processing method, which method includes: acquiring a first image; for any pixel point in the first image, using N (N≥2) searches associated with any one pixel point. table to determine the image block corresponding to any pixel point. The image block includes single or multiple pixel points, where the index area of each lookup table contains the any one pixel point, and the index areas of different lookup tables Different and at least partially the same; obtain a second image based on the image blocks corresponding to each pixel point in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; and output the second image.

这样，在图像处理过程中，对于每个像素点均使用多个查找表协同工作，从而在维持计算量和存储量线性增长的情况下扩增了感受野，提升了图像处理性能。相比之下，现有利用空域信息的查找表技术仅使用单个查找表，感受野的扩增会使得存储量指数增长，会受到存储器规模和硬件成本的限制。In this way, during the image processing process, multiple lookup tables are used to work together for each pixel, thereby expanding the receptive field and improving image processing performance while maintaining linear growth in the amount of calculation and storage. In contrast, existing lookup table technology that utilizes spatial domain information only uses a single lookup table. The expansion of the receptive field will increase the storage capacity exponentially and will be limited by memory scale and hardware cost.

在一种可能的实现方式中，利用N个与任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块，具体包括：将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，每个第一子图像块中均包括单个或多个像素点；将N个第一子图像块中相同位置的像素点均相加并求平均，以得到该任意一个像素点所对应的图像块。由于查找表对应的输出结果所对应的索引区域增加了，因此查找表的感受野也会相应的增加。同时，查找表的维度未发生变化，所以实现了在不增加查找表的维度的前提下，增大了查找表的感受野。In a possible implementation, N lookup tables associated with any one pixel are used to determine the image block corresponding to the any one pixel, which specifically includes: converting the index area of each lookup table in the N lookup tables Input it into the corresponding lookup table to obtain N first sub-image blocks. Each first sub-image block includes single or multiple pixels; the pixels at the same position in the N first sub-image blocks are evenly phased. Add and average to obtain the image block corresponding to any pixel. Since the index area corresponding to the output result of the lookup table increases, the receptive field of the lookup table will also increase accordingly. At the same time, the dimensions of the lookup table have not changed, so the receptive field of the lookup table can be increased without increasing the dimensions of the lookup table.

在一种可能的实现方式中，将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，具体包括：针对N个查找表中任意一个查找表的索引区域，将该任意一个查找表的索引区域以任意一个像素点为中心，并沿第一方向旋转M次，以得到M个索引区域，M≥1；基于M个索引区域中每个索引区域对应的查找表，确定M个索引区域各自所对应的第二子图像块，以得到M个第二子图像块，每个第二子图像块中均包括单个或多个像素点；基于M个索引区域各自所对应的旋转角度，分别对M个索引区域各自所对应的第二子图像块沿与第一方向相反的方向旋转相同的角度，以得到M个第三子图像块；将任意一个查找表的索引区域所对应的第一子图像块和M个第三子图像块中相同位置的像素点均相加并求平均，以及，将计算结果作为该任意一个查找表的索引区域所对应的第一子图像块。由此以通过对索引区域进行旋转的方式进一步在不增加查找表的维度的前提下，增大查找表的感受野。In one possible implementation, the index area of each lookup table in the N lookup tables is input into the corresponding lookup table to obtain N first sub-image blocks, which specifically includes: targeting any one of the N lookup tables For the index area of the lookup table, center the index area of any lookup table on any pixel and rotate it M times along the first direction to obtain M index areas, M≥1; based on each of the M index areas Lookup tables corresponding to the index areas determine the second sub-image blocks corresponding to each of the M index areas, so as to obtain M second sub-image blocks, each second sub-image block including single or multiple pixels; Based on the rotation angles corresponding to each of the M index areas, rotate the second sub-image blocks corresponding to each of the M index areas by the same angle in the direction opposite to the first direction to obtain M third sub-image blocks; Add and average the pixels at the same position in the first sub-image block and the M third sub-image blocks corresponding to the index area of any look-up table, and use the calculation result as the index of any look-up table The first sub-image block corresponding to the area. Thus, by rotating the index area, the receptive field of the lookup table is further increased without increasing the dimension of the lookup table.

在一种可能的实现方式中，N个查找表中第i个查找表的输出为第(i+1)个查找表的输入，N个查找表中第1个查找表的输入为第1个查找表的索引区域，N个查找表中第N个查找表的输出为该任意一个像素点所对应的图像块。也即是说，N个查找表可以级联，由此以实现增大查找表的感受野。In a possible implementation, the output of the i-th lookup table among the N lookup tables is the input of the (i+1)-th lookup table, and the input of the 1st lookup table among the N lookup tables is the 1st The index area of the lookup table, the output of the Nth lookup table among the N lookup tables is any image The image block corresponding to the prime point. That is to say, N lookup tables can be cascaded, thereby increasing the receptive field of the lookup tables.

在一种可能的实现方式中，N个查找表由M个查找表集合中所包含的查找表构成，每个查找表集合中均包括多个查找表。其中，第j个查找表集合中各个查找表的输入分别为以第(j-1)个查找表集合的输出为基准确定出的第j个查找表集合中相应的查找表的索引区域。第1个查找表集合中各个查找表的输入分别为以前述的任意一个像素点为基准确定出的第1个查找表集合中相应的查找表的索引区域。第M个查找表集合的输出为前述的任意一个像素点所对应的图像块。由此通过多个查找表互补并级联的方式确定出各个像素所对应的图像块，进一步地提升图像处理性能。In a possible implementation, N lookup tables are composed of lookup tables included in M lookup table sets, and each lookup table set includes multiple lookup tables. The inputs of each lookup table in the jth lookup table set are respectively the index areas of the corresponding lookup tables in the jth lookup table set determined based on the output of the (j-1)th lookup table set. The input of each lookup table in the first lookup table set is the index area of the corresponding lookup table in the first lookup table set determined based on any of the aforementioned pixels. The output of the M-th lookup table set is the image block corresponding to any of the aforementioned pixels. In this way, the image blocks corresponding to each pixel are determined through the complementary and cascaded manner of multiple lookup tables, further improving image processing performance.

第二方面，本申请提供一种图像处理装置，该装置包括：获取模块和处理模块。其中，获取模块用于获取第一图像。处理模块用于对于第一图像中的任意一个像素点，利用N个(N≥2)与该任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块，图像块中包括单个或多个像素点，其中，每个查找表的索引区域均包含该任意一个像素点，且不同的查找表的索引区域不同且至少有部分相同。处理模块还用于根据第一图像中各个像素点所对应的图像块，得到第二图像，其中，第二图像的分辨率高于或等于第一图像的分辨率；以及，输出第二图像。In a second aspect, this application provides an image processing device, which includes: an acquisition module and a processing module. Wherein, the acquisition module is used to acquire the first image. The processing module is configured to use N (N≥2) lookup tables associated with any pixel point in the first image to determine the image block corresponding to the any pixel point, and the image block includes Single or multiple pixels, wherein the index area of each lookup table contains any one pixel, and the index areas of different lookup tables are different and at least partially the same. The processing module is also configured to obtain a second image based on the image blocks corresponding to each pixel point in the first image, where the resolution of the second image is higher than or equal to the resolution of the first image; and output the second image.

在一种可能的实现方式中，处理模块在利用N个与任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块时，具体用于：将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，每个第一子图像块中均包括单个或多个像素点；将N个第一子图像块中相同位置的像素点均相加并求平均，以得到该任意一个像素点所对应的图像块。In a possible implementation, when the processing module uses N lookup tables associated with any one pixel point to determine the image block corresponding to the any one pixel point, it is specifically used to: convert each of the N lookup tables The index area of the lookup table is input into the corresponding lookup table, and N first sub-image blocks are obtained. Each first sub-image block includes single or multiple pixels; the same position in the N first sub-image blocks is obtained. The pixels are added together and averaged to obtain the image block corresponding to any pixel.

在一种可能的实现方式中，处理模块在将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块时，具体用于：针对N个查找表中任意一个查找表的索引区域，将该任意一个查找表的索引区域以任意一个像素点为中心，并沿第一方向旋转M次，以得到M个索引区域，M≥1；基于M个索引区域中每个索引区域对应的查找表，确定M个索引区域各自所对应的第二子图像块，以得到M个第二子图像块，每个第二子图像块中均包括多个像素点；基于M个索引区域各自所对应的旋转角度，分别对M个索引区域各自所对应的第二子图像块沿与第一方向相反的方向旋转相同的角度，以得到M个第三子图像块；将该任意一个查找表的索引区域所对应的第一子图像块和M个第三子图像块中相同位置的像素点均相加并求平均，以及，将计算结果作为该任意一个查找表的索引区域所对应的第一子图像块。In a possible implementation, when the processing module inputs the index area of each lookup table in the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, it is specifically used to: The index area of any lookup table in the lookup table is centered on any pixel and rotated M times along the first direction to obtain M index areas, M≥1; based on M The lookup table corresponding to each index area in the index areas determines the second sub-image block corresponding to each of the M index areas, so as to obtain M second sub-image blocks. Each second sub-image block includes multiple Pixel points; based on the rotation angles corresponding to each of the M index areas, rotate the second sub-image blocks corresponding to each of the M index areas by the same angle in the direction opposite to the first direction to obtain M third sub-image blocks. Image block; add and average the pixel points at the same position in the first sub-image block corresponding to the index area of any lookup table and the M third sub-image blocks, and use the calculation result as the any one Look up the first sub-image block corresponding to the index area of the table.

在一种可能的实现方式中，N个查找表中第i个查找表的输出为第(i+1)个查找表的输入，N个查找表中第1个查找表的输入为第1个查找表的索引区域，N个查找表中第N个查找表的输出为该任意一个像素点所对应的图像块。In a possible implementation, the output of the i-th lookup table among the N lookup tables is the input of the (i+1)-th lookup table, and the input of the 1st lookup table among the N lookup tables is the 1st The index area of the lookup table. The output of the Nth lookup table among the N lookup tables is the image block corresponding to any pixel.

在一种可能的实现方式中，N个查找表由M个查找表集合中所包含的查找表构成，每个查找表集合中均包括多个查找表。其中，第j个查找表集合中各个查找表的输入分别为以第(j-1)个查找表集合的输出为基准确定出的第j个查找表集合中相应的查找表的索引区域。第1个查找表集合中各个查找表的输入分别为以前述的任意一个像素点为基准确定出的第1个查找表集合中相应的查找表的索引区域。第M个查找表集合的输出为前述的任意一个像素点所对应的图像块。In a possible implementation, N lookup tables are composed of lookup tables included in M lookup table sets, and each lookup table set includes multiple lookup tables. The inputs of each lookup table in the jth lookup table set are respectively the index areas of the corresponding lookup tables in the jth lookup table set determined based on the output of the (j-1)th lookup table set. The input of each lookup table in the first lookup table set is the index area of the corresponding lookup table in the first lookup table set determined based on any of the aforementioned pixels. The output of the M-th lookup table set is the image block corresponding to any of the aforementioned pixels.

第三方面，本申请提供一种计算设备，包括：至少一个存储器，用于存储程序；至少一个处理器，用于执行存储器存储的程序；其中，当存储器存储的程序被执行时，处理器用于执行第一方面或第一方面的任一种可能的实现方式所描述的方法。In a third aspect, the present application provides a computing device, including: at least one memory for storing a program; at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to Perform the method described in the first aspect or any possible implementation of the first aspect.

第四方面，本申请提供一种计算设备集群，包括至少一个计算设备，每个计算设备均包括处理器和存储器。至少一个计算设备的处理器用于执行至少一个计算设备的存储器中存储的指令，以使得计算设备集群执行第一方面或第一方面的任一种可能的实现方式所描述的方法。In a fourth aspect, the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory. The processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device, so that the cluster of computing devices executes the method described in the first aspect or any possible implementation of the first aspect.

第五方面，本申请提供一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，当计算机程序在处理器上运行时，使得处理器执行第一方面或第一方面的任一种可能的实现方式所描述的方法。In a fifth aspect, the application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is run on a processor, it causes the processor to execute the first aspect or any one of the first aspects. Possible implementations of the methods described.

第六方面，本申请提供一种计算机程序产品，当计算机程序产品在处理器上运行时，使得处理器执行第一方面或第一方面的任一种可能的实现方式所描述的方法。In a sixth aspect, the present application provides a computer program product. When the computer program product is run on a processor, it causes the processor to execute the method described in the first aspect or any possible implementation of the first aspect.

可以理解的是，上述第二方面至第六方面的有益效果可以参见上述第一方面中的相关描述，在此不再赘述。It can be understood that the beneficial effects of the above-mentioned second to sixth aspects can be referred to the relevant descriptions in the above-mentioned first aspect, and will not be described again here.

附图说明 Description of the drawings

下面对实施例或现有技术描述中所需使用的附图作简单地介绍。The drawings required to be used in the description of the embodiments or prior art will be briefly introduced below.

图1是本申请实施例提供的一种不同复杂程度的CNN网络对图像进行处理的对比示意图；Figure 1 is a comparative schematic diagram of image processing by CNN networks of different complexity provided by the embodiment of the present application;

图2是本申请实施例提供的一种利用神经网络得到查找表的过程示意图；Figure 2 is a schematic diagram of a process for obtaining a lookup table using a neural network according to an embodiment of the present application;

图3是本申请实施例提供的一种图像处理***的架构示意图；Figure 3 is a schematic architectural diagram of an image processing system provided by an embodiment of the present application;

图4是本申请实施例提供的一种索引区域和等效索引区域的示意图；Figure 4 is a schematic diagram of an index area and an equivalent index area provided by an embodiment of the present application;

图5是本申请实施例提供的一种对索引区域进行旋转的示意图；Figure 5 is a schematic diagram of rotating an index area provided by an embodiment of the present application;

图6是本申请实施例提供的一种对查找表的输出结果进行旋转的示意图；Figure 6 is a schematic diagram of rotating the output result of the lookup table provided by the embodiment of the present application;

图7是本申请实施例提供的一种层级式表值重索引示意图；Figure 7 is a schematic diagram of a hierarchical table value re-indexing provided by an embodiment of the present application;

图8是本申请实施例提供的一种量化可感知的层级式神经网络训练示意图；Figure 8 is a schematic diagram of a quantitatively perceptible hierarchical neural network training provided by an embodiment of the present application;

图9是本申请实施例提供的一种多查找表协同工作的示意图；Figure 9 is a schematic diagram of a multi-lookup table collaborative work provided by an embodiment of the present application;

图10是本申请实施例提供的一种卷积网络对Y型索引模式的适配过程示意图；Figure 10 is a schematic diagram of the adaptation process of a convolutional network to the Y-shaped index mode provided by the embodiment of the present application;

图11是本申请实施例提供的一种对拜尔像素的图像进行处理的过程示意图；Figure 11 is a schematic diagram of a process for processing Bayer pixel images provided by an embodiment of the present application;

图12是本申请实施例提供的一种图像处理方法的流程示意图；Figure 12 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

图13是本申请实施例提供的一种图像处理装置的结构示意图；Figure 13 is a schematic structural diagram of an image processing device provided by an embodiment of the present application;

图14是本申请实施例提供的一种计算设备的结构示意图；Figure 14 is a schematic structural diagram of a computing device provided by an embodiment of the present application;

图15是本申请实施例提供的一种计算设备集群的结构示意图。Figure 15 is a schematic structural diagram of a computing device cluster provided by an embodiment of the present application.

具体实施方式Detailed ways

本文中术语“和/或”，是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。本文中符号“/”表示关联对象是或者的关系，例如A/B表示A或者B。The term "and/or" in this article is an association relationship that describes related objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. these three situations. The symbol "/" in this article indicates that the associated object is or, for example, A/B means A or B.

本文中的说明书和权利要求书中的术语“第一”和“第二”等是用于区别不同的对象，而不是用于描述对象的特定顺序。例如，第一响应消息和第二响应消息等是用于区别不同的响应消息，而不是用于描述响应消息的特定顺序。The terms "first", "second", etc. in the description and claims herein are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first response message and the second response message are used to distinguish different response messages, but are not used to describe a specific sequence of response messages.

在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "such as" in the embodiments of the present application is not to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary" or "such as" is intended to present the concept in a concrete manner.

在本申请实施例的描述中，除非另有说明，“多个”的含义是指两个或者两个以上，例如，多个处理单元是指两个或者两个以上的处理单元等；多个元件是指两个或者两个以上的元件等。In the description of the embodiments of this application, unless otherwise stated, the meaning of “multiple” refers to two or more, for example, multiple processing units refers to two or more processing units, etc.; multiple Component refers to two or more components, etc.

示例性的，由于复杂的神经网络的感受野(receptive field，RF)较大，因此可以提升神经网络的图像处理性能。例如，如图1所示，图1的(A)是神经网络的输入；图1的(B)是当神经网络为简单的CNN网络，且其感受野为17x17时，该神经网络输出的结果；图1的(C)是当神经网络为复杂的CNN网络，且其感受野为45x45时，该神经网络输出的结果；图1的(D)是真值，即原始图像。通过图1的(B)、(C)和(D)进行对比可知，复杂的神经网络由于具有较大的感受野，因此其能够产生与真值更加接近的结果，即其图像处理性能更优。For example, since the receptive field (RF) of a complex neural network is large, the image processing performance of the neural network can be improved. For example, as shown in Figure 1, (A) in Figure 1 is the input of the neural network; (B) in Figure 1 is the output result of the neural network when the neural network is a simple CNN network and its receptive field is 17x17 ; Figure 1 (C) is the result of the neural network output when the neural network is a complex CNN network and its receptive field is 45x45; Figure 1 (D) is the true value, that is, the original image. Comparing (B), (C) and (D) in Figure 1, we can see that complex neural networks have larger receptive fields, so they can produce results that are closer to the true value, that is, their image processing performance is better. .

一般地，可以将神经网络转换为查找表(look-up table，LUT)，并将查找表应用在图像处理中，以满足在终端设备上的部署。具体地，可以先利用训练完毕的神经网络获取到输入数据与输出数据间的映射关系，并以键-值的形式存储在查找表中。然后，在利用该查找表进行图像处理。例如，在获取到一个输入数据后，可以利用该输入数据在查找表中查找相应的数值结果，并输出该结果。由此即完成图像处理。Generally, the neural network can be converted into a look-up table (LUT), and the look-up table is applied in image processing to meet the deployment on the terminal device. Specifically, you can first use the trained neural network to obtain the mapping relationship between input data and output data, and store it in the lookup table in the form of key-value. Then, use this lookup table for image processing. For example, after obtaining an input data, you can use the input data to find the corresponding numerical result in the lookup table and output the result. This completes image processing.

举例来说，如图2的(a)所示，可以先对用于图像处理的神经网络进行训练，该神经网络可以由若干卷积层构成，输入与输出为单通道图像；与一般的图像处理的神经网络不同，该网络的感受野被严格限制在长宽为2x2的输入范围内，即一组输出结果仅取决于对应的4维输入I₀,I₁,I₂,I₃。该神经网络可以采用常规的反向梯度传播实现参数优化。For example, as shown in Figure 2 (a), a neural network for image processing can be trained first. The neural network can be composed of several convolutional layers, and the input and output are single-channel images; unlike general images The neural network processed is different. The receptive field of the network is strictly limited to the input range of length and width 2x2, that is, a set of output results only depends on the corresponding 4-dimensional input I ₀ , I ₁ , I ₂ , I ₃ . This neural network can use conventional backward gradient propagation to achieve parameter optimization.

在训练完成后，可以进行从神经网络到查找表的转换。如图2的(b)所示，在完成网络参数的训练后，通过遍历4维像素经过采样的数值组合来构造神经网络的输入，经过运算得到相应的输出结果。然后再将遍历得到的结果以键-值的形式存储在查找表中，得到4维的查找表，即4DLUT。After training is complete, the conversion from neural network to lookup table can be performed. As shown in (b) of Figure 2, after completing the training of network parameters, the input of the neural network is constructed by traversing the sampled numerical combination of 4-dimensional pixels, and the corresponding output result is obtained after operation. Then the traversal results are stored in the lookup table in the form of key-value, resulting in a 4-dimensional lookup table, namely 4DLUT.

最后，可以利用转换的查找表进行图像处理。如图2的(c)所示，给定一组待处理的且以I₀为中心点的输入I₀,I₁,I₂,I₃，查找已存储在表中的数值结果。若4DLUT中未直接存储采样点I₀,I₁,I₂,I₃对应的结果，则先获取与输入相邻采样点的键值，再通过插值操作计算输出结果。由此，即完成图像处理。示例性的，继续参阅图2的(c)，当需要对一个图像中的像素点I₀进行处理时，输入可以I₀,I₁,I₂,I₃，接着可以得到输出的像素点为即得到I₀所对应的图像块；在得到该图像中各个像素点对应的图像块后，将这些图像块分别置于它们各自所对应的像素点的位置，即可以得到处理后的图像。在一些实施例中，一个图像块中可以包括一个像素点，也可以包括多个像素点，具体可根据实际情况而定。Finally, the transformed lookup table can be utilized for image processing. As shown in (c) of Figure 2, given a set of inputs I ₀ , I ₁ , I ₂ , and I ₃ to be processed with I ₀ as the center point, find the numerical results stored in the table. If the results corresponding to sampling points I ₀ , I ₁ , I ₂ and I ₃ are not directly stored in 4DLUT, then First obtain the key value of the sampling point adjacent to the input, and then calculate the output result through interpolation operation. Thus, the image processing is completed. For example, continue to refer to (c) of Figure 2. When it is necessary to process the pixel point I ₀ in an image, the input can be I ₀ , I ₁ , I ₂ , I ₃ , and then the output pixel point can be obtained as That is, the image block corresponding to I ₀ is obtained; after obtaining the image block corresponding to each pixel point in the image, these image blocks are placed at the positions of their respective corresponding pixel points, and the processed image can be obtained. In some embodiments, an image block may include one pixel or multiple pixels, depending on the actual situation.

这种方式虽然可以在终端设备上部署，但查找表所需的存储大小常随着神经网络的感受野的线性增加而指数增加。以4DLUT为例，对于经过4比特(bit)采样的图像，4DLUT共需要(2⁴+1)⁴＝1.274MB的空间存储图像处理结果。而当LUT维度扩增1维，5DLUT则需要(2⁴+1)⁵＝21.665MB的存储空间。因此，当查找表的维度增加时，对终端设备的存储空间的要求也会增加，致使难以在存储器规模较小的终端设备上部署。另外，单个查找表的感受野(即索引范围)的大小受到存储器规模和硬件成本的限制，因此只能构建索引范围较小的查找表，这导致了图像处理的性能受限。Although this method can be deployed on terminal devices, the storage size required for the lookup table often increases exponentially with the linear increase of the receptive field of the neural network. Taking 4DLUT as an example, for a 4-bit sampled image, 4DLUT requires a total of (2 ⁴ +1) ⁴ =1.274MB of space to store the image processing results. When the LUT dimension is expanded by 1 dimension, 5DLUT requires (2 ⁴ +1) ⁵ = 21.665MB of storage space. Therefore, when the dimension of the lookup table increases, the requirements for the storage space of the terminal device will also increase, making it difficult to deploy on a terminal device with a small memory size. In addition, the size of the receptive field (i.e., index range) of a single lookup table is limited by the memory scale and hardware cost, so only a lookup table with a smaller index range can be constructed, which limits the performance of image processing.

有鉴于此，本申请实施例提供了一种图像处理方法，可以在图像处理过程中不增加查找表的维度的前提下，增加查找表的感受野的大小，显著缓解了终端设备上存储器规模和硬件成本的约束，从而在保持查找表方法高效低功耗特点的同时，大幅提升其在多种图像处理任务上的性能。In view of this, embodiments of the present application provide an image processing method that can increase the size of the receptive field of the lookup table without increasing the dimension of the lookup table during the image processing process, significantly alleviating the memory size and cost of the terminal device. Constraints on hardware costs, thereby greatly improving its performance on a variety of image processing tasks while maintaining the high efficiency and low power consumption of the lookup table method.

示例性的，图3示出了一种图像处理***的架构示意图。如图3所示，该图像处理***300可以包括：基于空域互补索引的多表协同处理单元310和/或基于表值重索引的多层级处理单元320。By way of example, Figure 3 shows a schematic architectural diagram of an image processing system. As shown in FIG. 3 , the image processing system 300 may include: a multi-table collaborative processing unit 310 based on spatial complementary indexing and/or a multi-level processing unit 320 based on table value re-indexing.

其中，基于空域互补索引的多表协同处理单元310主要是用于对于输入的图像中的任意一处局部区域，可以定义该区域的中心像素点为I₀，并围绕中心点I₀构造多个互补的索引区域，其中，每个索引区域中均包括中心点I₀。以及，基于构造出的多个互补的索引区域，确定出各个索引区域对应的查找表。示例性的，可以将构造出的各个索引区域输入至预先训练好的神经网络模型中，以得到各个索引区域所对应的查找表。示例性的，本实施例中描述的互补与数学定义上的严格定义不同，本实施例中描述的互补是各索引区域可以有重叠的部分。本实施例中描述的互补仅要求各查找表具有其余查找表索引无法覆盖到的区域。另外，基于空域互补索引的多表协同处理单元310还可以基于各个像素点所关联的各个索引区域对应的查找表，确定出各个像素点所对应的图像块，以完成图像处理。Among them, the multi-table collaborative processing unit 310 based on the spatial complementary index is mainly used to define the center pixel point of the area as I ₀ for any local area in the input image, and construct multiple tables around the center point I ₀ Complementary index regions, wherein each index region includes the center point I ₀ . And, based on the constructed multiple complementary index areas, determine the lookup table corresponding to each index area. For example, each constructed index area can be input into a pre-trained neural network model to obtain a lookup table corresponding to each index area. For example, the complementarity described in this embodiment is different from the strict mathematical definition. The complementarity described in this embodiment means that each index area may have overlapping parts. The complementation described in this embodiment only requires that each lookup table has an area that cannot be covered by the indexes of other lookup tables. In addition, the multi-table cooperative processing unit 310 based on the spatial complementary index can also determine the image block corresponding to each pixel point based on the lookup table corresponding to each index area associated with each pixel point to complete image processing.

对于围绕中心点I₀构造多个互补的索引区域，以构造一个索引区域为例，可以由预先设定的查找表的维度，确定索引区域中所包含的像素点的数量。例如，当查找表的维度为4维时，索引区域中则包含4个像素点，且这4个像素点中包含中心点I₀。For constructing multiple complementary index areas around the center point I ₀ , taking the construction of an index area as an example, the number of pixels contained in the index area can be determined by the dimensions of the preset lookup table. For example, when the dimension of the lookup table is 4, the index area contains 4 pixels, and these 4 pixels include the center point I ₀ .

对于确定查找表的过程，以确定4维的查找表，且输入图像中的一处区域为例，如图4所示，可以定义该区域的中心像素点为I₀。然后，围绕中心点I₀构造多个互补的索引，即LUT⁽¹⁾索引区域、LUT⁽²⁾索引区域、与LUT⁽³⁾索引区域。其中，LUT⁽¹⁾索引区域为[I₀][I₁][I₃][I₄]，LUT⁽²⁾索引区域为[I₀][I₂][I₆][I₈]，LUT⁽³⁾索引区域为[I₀][I₄][I₅][I₇]。在图4中，仅有LUT⁽¹⁾索引区域包含{I₁,I₃}，仅有LUT⁽²⁾索引区域包含{I₂,I₆,I₈}，仅有LUT⁽³⁾索引区域包含{I₅,I₇}，因此，各查找表对应的索引区域可以相互补充，以扩大感受野。当然，也可以选用其他的互补索引的查找表，不同的查找表间可以相互互补即可，此处不做限定。在确定出多个互补的索引区域后，可以将这些索引区域分别输入至预先训练好的神经网络模型中，以由该神经网络模型输出各个索引区域对应的查找表。For the process of determining the lookup table, a 4-dimensional lookup table is determined. Taking an area in the input image as an example, as shown in Figure 4, the center pixel of the area can be defined as I ₀ . Then, multiple complementary indexes are constructed around the center point I ₀ , namely, the LUT ⁽¹⁾ index area, the LUT ⁽²⁾ index area, and the LUT ⁽³⁾ index area. Among them, the LUT ⁽¹⁾ index area is [I ₀ ][I ₁ ][I ₃ ][I ₄ ], the LUT ⁽²⁾ index area is [I ₀ ][I ₂ ][I ₆ ][I ₈ ], The LUT ⁽³⁾ index area is [I ₀ ][I ₄ ][I ₅ ][I ₇ ]. In Figure 4, only the LUT ⁽¹⁾ index area contains {I ₁ , I ₃ }, only the LUT ⁽²⁾ index area contains {I ₂ , I ₆ , I ₈ }, and only the LUT ⁽³⁾ index area contains Contains {I ₅ , I ₇ }, therefore, the index areas corresponding to each lookup table can complement each other to expand the receptive field. Of course, other complementary index lookup tables can also be used. Different lookup tables can be complementary to each other, and there is no limitation here. After multiple complementary index areas are determined, these index areas can be input into a pre-trained neural network model respectively, so that the neural network model outputs a lookup table corresponding to each index area.

在确定出多个互补的索引区域的查找表后，可以将中心点I₀在各个查找表中对应的输出结果进行平均计算，即为所需的结果，即得到中心点I₀所对应的图像块。例如，继续参阅图4，中心点I₀对应的输出结果V₀＝(LUT⁽¹⁾[I₀][I₁][I₃][I₄]+LUT⁽²⁾[I₀][I₂][I₆][I₈]+LUT⁽³⁾[I₀][I₄][I₅][I₇])/3。当输入为[I₀][I₁][I₃][I₄]时，输出结果所对应的索引区域可以等效于由[I₀][I₁][I₃][I₄][I₅][I₆][I₇][I₈]组成。由于查找表对应的输出结果所对应的索引区域增加了，因此查找表的感受野也会相应的增加。同时，查找表的维度未发生变化，所以实现了在不增加查找表的维度的前提下，增大了查找表的感受野。其中，当输入数据为[I₀][I₁][I₃][I₄]时，通过查找表即可以查找得到输出结果为V₀。对于其他维度的查找表，可以参考增加4维查找表的感受野的方式，此处不再赘述。After determining the lookup tables of multiple complementary index areas, the output results corresponding to the center point I ₀ in each lookup table can be averaged to obtain the required result, that is, the image corresponding to the center point I ₀ is obtained. piece. For example, continuing to refer to Figure 4, the output result V ₀ corresponding to the center point I ₀ = (LUT ⁽¹⁾ [I ₀ ][I ₁ ][I ₃ ][I ₄ ]+LUT ⁽²⁾ [I ₀ ][I ₂ ][I ₆ ][I ₈ ]+LUT ⁽³⁾ [I ₀ ][I ₄ ][I ₅ ][I ₇ ])/3. When the input is [I ₀ ][I ₁ ][I ₃ ][I ₄ ], the index area corresponding to the output result can be equivalent to [I ₀ ][I ₁ ][I ₃ ][I ₄ ][ I ₅ ][I ₆ ][I ₇ ][I ₈ ]. Since the index area corresponding to the output result of the lookup table increases, the receptive field of the lookup table will also increase accordingly. At the same time, the dimensions of the lookup table have not changed, so the receptive field of the lookup table can be increased without increasing the dimensions of the lookup table. Among them, when the input data is [I ₀ ][I ₁ ][I ₃ ][I ₄ ], the output result can be found as V ₀ through the lookup table. For lookup tables in other dimensions, you can refer to the method of increasing the receptive field of a 4-dimensional lookup table, which will not be described again here.

在一些实施例中，为了进一步在不增加查找表的维度的前提下，增大查找表的感受野，还可以对构建出的各个查找表所对应的索引区域进行旋转，以及基于中心点I₀在旋转前的索引区域对应的查找表的输出结果和旋转后的索引区域对应的查找表的输出结果，确定出中心点I₀在所对应的查找表最终的输出结果。具体地，针对图4中任意一个查找表，以LUT⁽¹⁾为例，如图5的(A)所示，可以先基于索引区域为[I₀][I₁][I₃][I₄]的查找表，确定出中心点I₀在该索引区域下对应的输出V_j(1)。然后，如图5的(B)所示，将图5的(A) 中的索引区域逆时针旋转90度，这时的索引区域变为了[I₀][I₁][I₉][I₁₀]，然后可以基于索引区域为[I₀][I₁][I₉][I₁₀]的查找表，确定出中心点I₀在该索引区域下对应的输出V_j(2)。接着，如图5的(C)所示，将图5的(B)中的索引区域逆时针旋转90度，这时的索引区域变为了[I₀][I₉][I₁₁][I₁₂]，然后可以基于索引区域为[I₀][I₉][I₁₁][I₁₂]的查找表，确定出中心点I₀在该索引区域下对应的输出V_j(3)。接着，如图5的(D)所示，将图5的(C)中的索引区域逆时针旋转90度，这时的索引区域变为了[I₀][I₃][I₁₂][I₁₃]，然后可以基于索引区域为[I₀][I₃][I₁₂][I₁₃]的查找表，确定出中心点I₀在该索引区域下对应的输出V_j(4)。由于V_j(2)和V_j(1)不在同一计算区域，为了统一，可以将V_j(2)顺时针旋转90度，以得到V_j(2)′。同样的，可以将V_j(3)顺时针旋转180度，以得到V_j(3)′，以及，将V_j(4)顺时针旋转270度，以得到V_j(4)′。例如，如图6的(A)所示，假设V_j(3)为[1][2][3][4]，将其顺时针旋转90度，可以得到如图6的(B)所示的结果；在图6的(B)的基础上再顺时针旋转90度，可以得到如图6的(C)所示的结果，其中，图6的(C)所示的结果即为所需的结果V_j(3)′。最后，可以将V_j(1)、V_j(2)′、V_j(3)′和V_j(4)′相加，并做平均，以及将计算得到的结果作为：当输入为[I₀][I₁][I₃][I₄]时通过LUT⁽¹⁾得到的输出结果，即V＝(V_j(1)+V_j(2)′+V_j(3)′+V_j(4)′)/4。对于增加其他的查找表的感受野的方式，可以按照图5中所描述的方式进行处理，此处不再赘述。本实施中，对于旋转的角度可根据实际情况进行选择，此处不做限定，例如，可以仅选择旋转90度，也可以仅选择旋转180度或者270度，亦可以选择旋转90度和180度，又可以选择旋转180度和270度，等等。同样的，在确定出中心点I₀在各个查找表中对应的输出结果后，可以将这些结果相加并求平均，以及计算结果作为最终所需的结果，以及可以在查找表中建立该输入和该输出间的关联关系，进而得到最新的查找表。示例性的，继续参阅图4，当选择对各个查找表均旋转90度、180度和270度时，最终得到的等效索引区域可以如图4中所示的等效索引区域。通过这种方式，实现了进一步增加查找表对应的输出结果所对应的索引区域，从而进一步地增加了查找表的感受野。In some embodiments, in order to further increase the receptive field of the lookup table without increasing the dimension of the lookup table, the index area corresponding to each constructed lookup table can also be rotated, and based on the center point I ₀ The output result of the lookup table corresponding to the index area before rotation and the output result of the lookup table corresponding to the index area after rotation determine the final output result of the lookup table corresponding to the center point I ₀ . Specifically, for any lookup table in Figure 4, taking LUT ⁽¹⁾ as an example, as shown in (A) of Figure 5, the index area can first be [I ₀ ][I ₁ ][I ₃ ][I ₄ ] lookup table to determine the output V _j (1) corresponding to the center point I ₀ in the index area. Then, as shown in (B) of Fig. 5, (A) of Fig. 5 The index area in is rotated 90 degrees counterclockwise. At this time, the index area becomes [I ₀ ][I ₁ ][I ₉ ][I ₁₀ ], and then based on the index area it can be [I ₀ ][I ₁ ][I ₉ ][I ₁₀ ] lookup table to determine the output V _j (2) corresponding to the center point I ₀ in the index area. Next, as shown in Figure 5(C), the index area in Figure 5(B) is rotated 90 degrees counterclockwise. The index area at this time becomes [I ₀ ][I ₉ ][I ₁₁ ][I ₁₂ ], and then based on the lookup table with the index area [I ₀ ][I ₉ ][I ₁₁ ][I ₁₂ ], the output V _j corresponding to the center point I ₀ under the index area can be determined (3). Next, as shown in Figure 5 (D), the index area in Figure 5 (C) is rotated 90 degrees counterclockwise. The index area at this time becomes [I ₀ ][I ₃ ][I ₁₂ ][I ₁₃ ], and then based on the lookup table with the index area [I ₀ ][I ₃ ][I ₁₂ ][I ₁₃ ], the output V _j corresponding to the center point I ₀ under the index area can be determined (4). Since V _j (2) and V _j (1) are not in the same calculation area, for unification, V _j (2) can be rotated 90 degrees clockwise to obtain V _j (2)′. Likewise, V _j (3) can be rotated 180 degrees clockwise to obtain V _j (3)', and V _j (4) can be rotated 270 degrees clockwise to obtain V _j (4)'. For example, as shown in Figure 6 (A), assuming that V _j (3) is [1][2][3][4], rotate it 90 degrees clockwise, you can get the figure shown in Figure 6 (B) The result shown in Figure 6(B); by rotating 90 degrees clockwise on the basis of Figure 6(B), the result shown in Figure 6(C) can be obtained, where the result shown in Figure 6(C) is the result The desired result is V _j (3)′. Finally, V _j (1), V _j (2)′, V _j (3)′, and V _j (4)′ can be added and averaged, and the calculated result is: When the input is [I ₀ ][I ₁ ][I ₃ ][I ₄ ] The output result obtained through LUT ⁽¹⁾ , that is, V=(V _j (1)+V _j (2)′+V _j (3)′+V _j (4)′)/4. The method of increasing the receptive fields of other lookup tables can be processed in the manner described in Figure 5, and will not be described again here. In this implementation, the angle of rotation can be selected according to the actual situation, and is not limited here. For example, you can only choose to rotate 90 degrees, you can only choose to rotate 180 degrees or 270 degrees, or you can choose to rotate 90 degrees and 180 degrees. , and you can choose to rotate 180 degrees and 270 degrees, etc. Similarly, after determining the output results corresponding to the center point I ₀ in each lookup table, these results can be added and averaged, and the calculated results can be used as the final required results, and the input can be established in the lookup table and the relationship between the output, and then obtain the latest lookup table. For example, continuing to refer to FIG. 4 , when each lookup table is rotated by 90 degrees, 180 degrees, and 270 degrees, the final equivalent index area may be the equivalent index area as shown in FIG. 4 . In this way, the index area corresponding to the output result corresponding to the lookup table is further increased, thereby further increasing the receptive field of the lookup table.

基于表值重索引的多层级处理单元320主要是用于对以某个中心点I₀相关的多个查找表进行层级处理，其中，第k级查找表的输入是第(k-1)级查找表的输出。由此通过级联多个查找表间的输入与输出，来扩大等效感受野。示例性的，如图7所示，以两级查找表的级联过程为例，LUT⁽¹⁾表示首级查找表，LUT⁽²⁾表示次级查找表。在图7中，可以将LUT⁽¹⁾中的索引取值结果用作次级LUT⁽²⁾的索引，这种层级式的索引过程可以表示为：
V＝LUT⁽²⁾[LUT⁽¹⁾[I_*]][LUT⁽¹⁾[I_*]][LUT⁽¹⁾[I_*]][LUT⁽¹⁾[I_*]]The multi-level processing unit 320 based on table value re-indexing is mainly used to perform hierarchical processing on multiple lookup tables related to a certain center point I ₀ , where the input of the k-th level lookup table is the (k-1)-th level The output of the lookup table. This expands the equivalent receptive field by cascading the inputs and outputs between multiple lookup tables. Illustratively, as shown in Figure 7, taking the cascade process of two-level lookup tables as an example, LUT ⁽¹⁾ represents the first-level lookup table, and LUT ⁽²⁾ represents the secondary lookup table. In Figure 7, the index value result in LUT ⁽¹⁾ can be used as the index of the secondary LUT ⁽²⁾ . This hierarchical indexing process can be expressed as:
V＝LUT ⁽²⁾ [LUT ⁽¹⁾ [I _* ]][LUT ⁽¹⁾ [I _* ]][LUT ⁽¹⁾ [I _* ]][LUT ⁽¹⁾ [I _* ]]

当将低质量的图像输入到基于表值重索引的多层级处理单元320中后，通过基于表值重索引的多层级处理单元320的层级处理，即可以输出高质量的图像。When a low-quality image is input into the multi-level processing unit 320 based on table value re-indexing, a high-quality image can be output through the hierarchical processing of the multi-level processing unit 320 based on table value re-indexing.

从感受野的角度来看，这种级联框架在原理上类似于在深度卷积神经网络中级联多个卷积层。但与CNN中的卷积层级联不同，由于LUT大小的限制，图像数据的索引通常以整型数的格式存储。而在前述图2中描述的从神经网络到查找表的转换步骤之前，训练神经网络需要浮点(Float-Point)数据类型的梯度。为了解决数据类型带来的问题，本实施例中设计了一种量化可感知的重索引机制，将层级式索引的模式引入到神经网络的训练过程中。如图8所示，LUT⁽¹⁾的输出值在前向传递中被量化为整数(即指向Integer的虚线所指的方向)，而其梯度在反向传递中被保留为浮点值(即指向Float-Point的虚线所指的方向)。这样，数值量化操作在神经网络的层级式训练过程中可以被感知与建模，从而弥补由数值量化引起的性能损失。From the perspective of receptive fields, this cascade framework is similar in principle to cascading multiple convolutional layers in a deep convolutional neural network. However, unlike the cascade of convolutional layers in CNN, due to the limitation of LUT size, the index of image data is usually stored in the format of integer numbers. Before the conversion step from a neural network to a lookup table described in Figure 2, training the neural network requires gradients of the Float-Point data type. In order to solve the problems caused by data types, a quantification-aware re-indexing mechanism is designed in this embodiment to introduce the hierarchical indexing pattern into the training process of the neural network. As shown in Figure 8, the output value of LUT ⁽¹⁾ is quantized to an integer in the forward pass (i.e., the direction pointed by the dashed line pointing to Integer), while its gradient is retained as a floating point value in the backward pass (i.e. points in the direction of the dashed line pointing to the Float-Point). In this way, numerical quantization operations can be perceived and modeled during the hierarchical training process of neural networks, thus making up for the performance loss caused by numerical quantization.

在一些实施例中，基于空域互补索引的多表协同处理单元310和基于表值重索引的多层级处理单元320，可以单独使用，也可以组合在一起使用。示例性的，图9示出了一种两者组合在一起使用的过程。如图9所示，将多查找表(Multiple LUT)简写为MuLUT，MuLUT-S/D/Y分别表示三种具有不同像素索引模式的LUT。Block 1x1表示输出图像的分辨率与输入图像保持相同，Block 2x2表示输出图像的分辨率为输入图像的2倍。三种LUT对应的神经网络均由若干卷积层堆叠连接实现，其中S型LUT对应的卷积网络首层采用卷积核大小为2x2的标准卷积，记为Conv2x2。D型LUT对应的卷积网络首层采用卷积核大小为2x2的空洞卷积(dilated convolution)，记为Conv2x2-D2。Y型LUT对应的卷积网络首层采用卷积核大小为1x4的标准卷积实现，记为Conv1x4。由于Y型索引模式不属于常规的网格化(grid)索引，无法直接调用现有的函数库进行运算。因此对应卷积网络的运算需要进行特殊适配。如图10所示，首先，将输入图像以滑动窗口的方式拆分为3x3大小的图像块，即unfold操作。然后在得到的各个图像块中，选取Y型索引模式对应的像素坐标，并将相应坐标下的数值重排列为1x4的图像块。经过拆分与重排列后的图像即可适配首层为Conv1x4的卷积网络。 In some embodiments, the multi-table collaborative processing unit 310 based on spatial complementary index and the multi-level processing unit 320 based on table value re-indexing can be used alone or in combination. By way of example, Figure 9 shows a process in which the two are combined together. As shown in Figure 9, multiple lookup tables (Multiple LUT) are abbreviated as MuLUT, and MuLUT-S/D/Y respectively represent three LUTs with different pixel index modes. Block 1x1 means that the resolution of the output image remains the same as that of the input image, and Block 2x2 means that the resolution of the output image is 2 times that of the input image. The neural networks corresponding to the three LUTs are all implemented by stacking and connecting several convolutional layers. The first layer of the convolutional network corresponding to the S-type LUT uses a standard convolution with a convolution kernel size of 2x2, which is recorded as Conv2x2. The first layer of the convolutional network corresponding to the D-type LUT uses a dilated convolution with a convolution kernel size of 2x2, which is recorded as Conv2x2-D2. The first layer of the convolutional network corresponding to the Y-type LUT is implemented using a standard convolution with a convolution kernel size of 1x4, which is recorded as Conv1x4. Since the Y-shaped index mode does not belong to the conventional grid index, the existing function library cannot be directly called for operation. Therefore, the operation of the corresponding convolutional network requires special adaptation. As shown in Figure 10, first, the input image is split into image blocks of 3x3 size in a sliding window, which is the unfold operation. Then, in each obtained image block, select the pixel coordinates corresponding to the Y-type index pattern, and rearrange the values at the corresponding coordinates into 1x4 image blocks. The split and rearranged image can be adapted to a convolutional network with a Conv1x4 first layer.

继续参阅图9，在图9中，首先选用3个与中心点I₀相关的多个查找表对应的索引范围，然后分别将各个索引范围中的数据输入至各自对应的卷积网络中，它们各自对应的卷积网络输出的结果均为1维数据。接着，将各个卷积网络的输出结果合并为一个，例如求平均值等方式。然后，在将该结果作为中心点，并确定出与该结果相关的多个查找表的索引范围，以及分别将各个索引范围中的数据输入至各自对应的卷积网络中，它们各自对应的卷积网络输出的结果均为与所需的查找表的维度相同的维度的数据。最后，可以将这些数据合并为一个，并建立其与某个包含中心点I₀的索引范围间的关联关系，由此即建立出新的查找表。该查找表的感受野大于之前的查找表的感受野，且维度不变。Continue to refer to Figure 9. In Figure 9, first select three index ranges corresponding to multiple lookup tables related to the center point I ₀ , and then input the data in each index range into their corresponding convolutional networks. The output results of the corresponding convolutional networks are all 1-dimensional data. Then, the output results of each convolutional network are combined into one, such as averaging. Then, using the result as the center point, determining the index ranges of multiple lookup tables related to the result, and inputting the data in each index range into their corresponding convolutional networks, their respective corresponding volumes The output results of the product network are all data with the same dimensions as the required lookup table dimensions. Finally, these data can be merged into one and associated with an index range containing the center point I ₀ to establish a new lookup table. The receptive field of this lookup table is larger than that of the previous lookup table, and the dimensions remain unchanged.

在一些实施例中，当输入的图像的像素是拜尔(bayer)像素排列模式时，可以以2倍的步长滑动窗口，在2x2的窗口内对输入进行LUT查询，从而避免bayer像素模式的影响。然后，在采用前述的所描述的图像处理***300进行多查找表协同处理，和/或，将第一阶段查找表的取值作为第二阶段的输入，实现层级式处理。示例性的，如图11所示，可以先以2倍步长，分别提取出红色通道的像素点、绿色通道的像素点和蓝色通道的像素点，然后，在采用前述的所描述的图像处理***300分别对提取出的像素点进行处理。In some embodiments, when the pixels of the input image are arranged in a Bayer pixel pattern, the input can be LUT queried within a 2x2 window by sliding the window with a step size of 2 times, thereby avoiding the Bayer pixel pattern. Influence. Then, the aforementioned image processing system 300 is used to perform multi-lookup table collaborative processing, and/or the value of the first-stage lookup table is used as the input of the second stage to implement hierarchical processing. For example, as shown in Figure 11, you can first extract the pixels of the red channel, the pixels of the green channel, and the pixels of the blue channel with a step size of 2 times, and then use the previously described image. The processing system 300 processes the extracted pixels respectively.

接下来，基于前述所描述的内容，对本申请实施例中提供的图像处理方法进行介绍。Next, based on the foregoing description, the image processing method provided in the embodiment of the present application is introduced.

示例性的，图12示出了一种图像处理方法的流程。可以理解，该方法可以通过任何具有计算、处理能力的装置、设备、平台、设备集群来执行。如图12所述，该图像处理方法可以包括以下步骤：By way of example, Figure 12 shows the flow of an image processing method. It can be understood that this method can be executed by any device, device, platform, or device cluster with computing and processing capabilities. As shown in Figure 12, the image processing method may include the following steps:

S1201、获取待处理的第一图像。S1201. Obtain the first image to be processed.

本实施例中，当用户输入其所需处理的图像后，就可以获取到待处理的第一图像。另外，也可以从用户设定的存储区域中自动获取待处理的第一图像。具体可根据实际情况而定。In this embodiment, after the user inputs the image that needs to be processed, the first image to be processed can be obtained. In addition, the first image to be processed can also be automatically obtained from a storage area set by the user. The details may be determined according to the actual situation.

S1202、对于第一图像中的任意一个像素点，均利用N个(N≥2)与该任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块，该图像块中包括单个或多个像素点，其中，每个查找表的索引区域均包含该任意一个像素点，且不同的查找表的索引区域不同且至少有部分相同。S1202. For any pixel in the first image, use N (N≥2) lookup tables associated with the pixel to determine the image block corresponding to the pixel. The image block includes Single or multiple pixels, wherein the index area of each lookup table contains any one pixel, and the index areas of different lookup tables are different and at least partially the same.

本实施例中，对于第一图像中的任意一个像素点，均可以利用多个与该任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块。其中，该图像块中包括单个或多个像素点。另外，每个查找表的索引范围均包含该任意一个像素点，且不同的第一查找表的索引范围不同且至少有部分相同，即不同的查找表的索引范围互补。举例来说，继续参阅图4，该任意一个像素点可以为中心点I₀，即LUT⁽¹⁾、LUT⁽²⁾和LUT⁽³⁾可以为与中心点I₀关联的查找表。利用多个与该任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块的方式，可以参见前述“基于空域互补索引的多表协同处理单元310得到中心点I₀所对应的图像块”的过程，此处不再赘述。In this embodiment, for any pixel in the first image, multiple lookup tables associated with the pixel can be used to determine the image block corresponding to the pixel. The image block includes single or multiple pixels. In addition, the index range of each lookup table includes any one pixel, and the index ranges of different first lookup tables are different and at least partially the same, that is, the index ranges of different lookup tables are complementary. For example, continuing to refer to FIG. 4 , any pixel point can be the center point I ₀ , that is, LUT ⁽¹⁾ , LUT ⁽²⁾ and LUT ⁽³⁾ can be lookup tables associated with the center point I ₀ . For a method of using multiple lookup tables associated with any one pixel point to determine the image block corresponding to any one pixel point, please refer to the aforementioned "Multi-table collaborative processing unit 310 based on spatial complementary index to obtain the center point I ₀ corresponding to The process of "image block" will not be described again here.

作为一种可能实现方式，可以将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块。其中，每个第一子图像块中均包括单个或多个像素点。然后，在将N个第一子图像块中相同位置的像素点均相加并求平均，以得到该任意一个像素点所对应的图像块。由于查找表对应的输出结果所对应的索引区域增加了，因此查找表的感受野也会相应的增加。同时，查找表的维度未发生变化，所以实现了在不增加查找表的维度的前提下，增大了查找表的感受野。详见前述对图4的描述，此处不再赘述。As a possible implementation, the index area of each lookup table in the N lookup tables can be input into the corresponding lookup table to obtain N first sub-image blocks. Each first sub-image block includes single or multiple pixels. Then, the pixel points at the same position in the N first sub-image blocks are added and averaged to obtain the image block corresponding to any one pixel point. Since the index area corresponding to the output result of the lookup table increases, the receptive field of the lookup table will also increase accordingly. At the same time, the dimensions of the lookup table have not changed, so the receptive field of the lookup table can be increased without increasing the dimensions of the lookup table. For details, please refer to the aforementioned description of Figure 4 and will not be described again here.

进一步地，针对N个查找表中任意一个查找表的索引区域，可以将任意一个查找表的索引区域以任意一个像素点为中心，并沿第一方向旋转M次，以得到M个索引区域，M≥1。然后，在基于M个索引区域中每个索引区域对应的查找表，确定M个索引区域各自所对应的第二子图像块，以得到M个第二子图像块。其中，每个第二子图像块中均包括单个或多个像素点。接着，可以基于M个索引区域各自所对应的旋转角度，分别对M个索引区域各自所对应的第二子图像块沿与第一方向相反的方向旋转相同的角度，以得到M个第三子图像块。最后，将任意一个查找表的索引区域所对应的第一子图像块和M个第三子图像块中相同位置的像素点均相加并求平均，以及，将计算结果作为任意一个查找表的索引区域所对应的第一子图像块。由此以通过对索引区域进行旋转的方式进一步在不增加查找表的维度的前提下，增大查找表的感受野。详见前述有关图5和图6的描述，此处不再赘述。Further, for the index area of any one of the N look-up tables, the index area of any one look-up table can be centered on any pixel and rotated M times along the first direction to obtain M index areas, M≥1. Then, based on the lookup table corresponding to each index area in the M index areas, the second sub-image blocks corresponding to each of the M index areas are determined to obtain M second sub-image blocks. Each second sub-image block includes single or multiple pixels. Then, based on the rotation angles corresponding to each of the M index regions, the second sub-image blocks corresponding to each of the M index regions can be rotated by the same angle in the direction opposite to the first direction to obtain M third sub-image blocks. Image blocks. Finally, the pixel points at the same position in the first sub-image block and the M third sub-image blocks corresponding to the index area of any look-up table are added and averaged, and the calculation result is used as the result of any look-up table. The first sub-image block corresponding to the index area. Thus, by rotating the index area, the receptive field of the lookup table is further increased without increasing the dimension of the lookup table. For details, please refer to the aforementioned descriptions of Figures 5 and 6 and will not be described again here.

作为又一种可能的实现方式，N个查找表中第i个查找表的输出为第(i+1)个查找表的输入，N个查找表中第1个查找表的输入为第1个查找表的索引区域，N个查找表中第N个查找表的输出为任意一个像素点所对应的图像块。也即是说，N个查找表可以级联，由此以实现增大查找表的感受野。 As another possible implementation, the output of the i-th lookup table among the N lookup tables is the input of the (i+1)-th lookup table, and the input of the 1st lookup table among the N lookup tables is the 1st The index area of the lookup table. The output of the Nth lookup table among the N lookup tables is the image block corresponding to any pixel. That is to say, N lookup tables can be cascaded, thereby increasing the receptive field of the lookup tables.

作为另一种可能的实现方式，N个查找表可以由M个查找表集合中所包含的查找表构成，每个查找表集合中均包括多个查找表。其中，第j个查找表集合中各个查找表的输入分别为以第(j-1)个查找表集合的输出为基准确定出的第j个查找表集合中相应的查找表的索引区域。第1个查找表集合中各个查找表的输入分别为以前述的任意一个像素点为基准确定出的第1个查找表集合中相应的查找表的索引区域。第M个查找表集合的输出为前述的任意一个像素点所对应的图像块。示例性的，该处理过程可以理解是前述图9中的处理过程，详见前述描述，此处不再赘述。As another possible implementation, N lookup tables may be composed of lookup tables included in M lookup table sets, and each lookup table set includes multiple lookup tables. The inputs of each lookup table in the jth lookup table set are respectively the index areas of the corresponding lookup tables in the jth lookup table set determined based on the output of the (j-1)th lookup table set. The input of each lookup table in the first lookup table set is the index area of the corresponding lookup table in the first lookup table set determined based on any of the aforementioned pixels. The output of the M-th lookup table set is the image block corresponding to any of the aforementioned pixels. For example, this processing process can be understood as the aforementioned processing process in FIG. 9. For details, please refer to the aforementioned description and will not be described again here.

S1203、根据第一图像中各个像素点所对应的图像块，得到第二图像，第二图像的分辨率高于或等于第一图像的分辨率。S1203. Obtain a second image based on the image blocks corresponding to each pixel point in the first image. The resolution of the second image is higher than or equal to the resolution of the first image.

本实施例中，在得到第一图像中各个像素点所对应的图像块后，可以将这些图像块拼接在一起，以得到第二图像。其中，第二图像的分辨率高于或等于第一图像的分辨率。在一些实施例中，在拼接各个图像块时，可以将各个图像块置于它们各自所对应的像素点的位置处。例如，在第一图像中的像素点1与图像块1对应，那么可以将图像块1置于像素点1的位置处。In this embodiment, after obtaining the image blocks corresponding to each pixel point in the first image, these image blocks can be spliced together to obtain the second image. Wherein, the resolution of the second image is higher than or equal to the resolution of the first image. In some embodiments, when splicing each image block, each image block may be placed at the position of its corresponding pixel point. For example, if pixel point 1 in the first image corresponds to image block 1, then image block 1 can be placed at the position of pixel point 1.

S1204、输出第二图像。S1204. Output the second image.

本实施例中，在获取到第二图像后，可以输出该第二图像。比如显示在设备的显示屏上等。In this embodiment, after the second image is acquired, the second image can be output. For example, displayed on the device's display, etc.

由此，在图像处理过程中，对于每个像素点均使用多个查找表协同工作，从而在维持计算量和存储量线性增长的情况下扩增了感受野，提升了图像处理性能。相比之下，现有利用空域信息的查找表技术仅使用单个查找表，感受野的扩增会使得存储量指数增长，会受到存储器规模和硬件成本的限制。Therefore, during the image processing process, multiple lookup tables are used to work together for each pixel, thereby expanding the receptive field and improving image processing performance while maintaining linear growth in the amount of calculation and storage. In contrast, existing lookup table technology that utilizes spatial domain information only uses a single lookup table. The expansion of the receptive field will increase the storage capacity exponentially and will be limited by memory scale and hardware cost.

基于上述实施例中的方法，本申请实施例还提供了一种图像处理装置。Based on the methods in the above embodiments, embodiments of the present application also provide an image processing device.

示例性的，图13示出了一种图像处理装置。如图13所示，该图像处理装置1300可以包括获取模块1301和处理模块1302。其中，获取模块1301用于获取第一图像。处理模块1302用于对于第一图像中的任意一个像素点，利用N个(N≥2)与该任意一个像素点关联的查找表，确定该任意一个像素点所对应的图像块，图像块中包括单个或多个像素点，其中，每个查找表的索引区域均包含该任意一个像素点，且不同的查找表的索引区域不同且至少有部分相同。处理模块1302还用于根据第一图像中各个像素点所对应的图像块，得到第二图像，其中，第二图像的分辨率高于或等于第一图像的分辨率；以及，输出第二图像。By way of example, FIG. 13 shows an image processing device. As shown in Figure 13, the image processing device 1300 may include an acquisition module 1301 and a processing module 1302. Among them, the acquisition module 1301 is used to acquire the first image. The processing module 1302 is configured to use N (N≥2) lookup tables associated with any pixel point in the first image to determine the image block corresponding to the arbitrary pixel point. Including single or multiple pixels, wherein the index area of each lookup table contains any one pixel, and the index areas of different lookup tables are different and at least partially the same. The processing module 1302 is also configured to obtain a second image based on the image blocks corresponding to each pixel point in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; and, output the second image .

在一些实施例中，处理模块1302在利用N个与任意一个像素点关联的查找表，确定任意一个像素点所对应的图像块时，具体用于：将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，每个第一子图像块中均包括多个像素点；将N个第一子图像块中相同位置的像素点均相加并求平均，以得到任意一个像素点所对应的图像块。In some embodiments, when using N lookup tables associated with any one pixel point to determine the image block corresponding to any one pixel point, the processing module 1302 is specifically used to: The index area is input into the corresponding lookup table to obtain N first sub-image blocks. Each first sub-image block includes multiple pixels; the pixels at the same position in the N first sub-image blocks are evenly matched. Add and average to obtain the image block corresponding to any pixel.

在一些实施例中，处理模块1302在将N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块时，具体用于：针对N个查找表中任意一个查找表的索引区域，将该任意一个查找表的索引区域以任意一个像素点为中心，并沿第一方向旋转M次，以得到M个索引区域，M≥1；基于M个索引区域中每个索引区域对应的查找表，确定M个索引区域各自所对应的第二子图像块，以得到M个第二子图像块，每个第二子图像块中均包括单个或多个像素点；基于M个索引区域各自所对应的旋转角度，分别对M个索引区域各自所对应的第二子图像块沿与第一方向相反的方向旋转相同的角度，以得到M个第三子图像块；将该任意一个查找表的索引区域所对应的第一子图像块和M个第三子图像块中相同位置的像素点均相加并求平均，以及，将计算结果作为该任意一个查找表的索引区域所对应的第一子图像块。In some embodiments, when the processing module 1302 inputs the index area of each lookup table in the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, it is specifically used to: for the N lookup tables The index area of any lookup table is centered on any pixel and rotated M times along the first direction to obtain M index areas, M≥1; based on M indexes The lookup table corresponding to each index area in the area determines the second sub-image blocks corresponding to each of the M index areas to obtain M second sub-image blocks. Each second sub-image block includes a single or multiple Pixel points; based on the rotation angles corresponding to each of the M index areas, rotate the second sub-image blocks corresponding to each of the M index areas by the same angle in the direction opposite to the first direction to obtain M third sub-image blocks. Image block; add and average the pixel points at the same position in the first sub-image block corresponding to the index area of any lookup table and the M third sub-image blocks, and use the calculation result as the any one Look up the first sub-image block corresponding to the index area of the table.

在一些实施例中，N个查找表中第i个查找表的输出为第(i+1)个查找表的输入，N个查找表中第1个查找表的输入为第1个查找表的索引区域，N个查找表中第N个查找表的输出为任意一个像素点所对应的图像块。In some embodiments, the output of the i-th lookup table among the N lookup tables is the input of the (i+1)th lookup table, and the input of the 1st lookup table among the N lookup tables is the input of the 1st lookup table. In the index area, the output of the Nth lookup table among the N lookup tables is the image block corresponding to any pixel.

在一些实施例中，N个查找表由M个查找表集合中所包含的查找表构成，每个查找表集合中均包括多个查找表。其中，第j个查找表集合中各个查找表的输入分别为以第(j-1)个查找表集合的输出为基准确定出的第j个查找表集合中相应的查找表的索引区域。第1个查找表集合中各个查找表的输入分别为以任意一个像素点为基准确定出的第1个查找表集合中相应的查找表的索引区域。第M个查找表集合的输出为任意一个像素点所对应的图像块。In some embodiments, the N lookup tables are composed of lookup tables contained in M lookup table sets, each lookup table set including multiple lookup tables. The inputs of each lookup table in the jth lookup table set are respectively the index areas of the corresponding lookup tables in the jth lookup table set determined based on the output of the (j-1)th lookup table set. The input of each lookup table in the first lookup table set is the index area of the corresponding lookup table in the first lookup table set determined based on any pixel point. The output of the M-th lookup table set is the image block corresponding to any pixel.

应当理解的是，上述装置用于执行上述实施例中的方法，装置中相应的程序模块，其实现原理和技术效果与上述方法中的描述类似，该装置的工作过程可参考上述方法中的对应过程，此处不再赘述。It should be understood that the above device is used to execute the method in the above embodiment. The corresponding program module in the device has implementation principles and technical effects similar to those described in the above method. For the working process of the device, please refer to the corresponding program module in the above method. The process will not be described again here.

在一些实施例中，图13中所示的获取模块1301和处理模块1302均可以通过软件实现，或者可以通过硬件实现。示例性的，接下来以图13中所示的获取模块1301为例，介绍获取模块1301的实现方式。类似的，图13中所示的处理模块1302的实现方式可以参考图13中所示的获取模块1301的实现方式。In some embodiments, both the acquisition module 1301 and the processing module 1302 shown in Figure 13 can be implemented through software, or can be implemented through Implemented through hardware. Illustratively, the following takes the acquisition module 1301 shown in Figure 13 as an example to introduce the implementation of the acquisition module 1301. Similarly, the implementation of the processing module 1302 shown in FIG. 13 may refer to the implementation of the acquisition module 1301 shown in FIG. 13 .

模块作为软件功能单元的一种举例，处理模块1302可以包括运行在计算实例上的代码。其中，计算实例可以包括物理主机(计算设备)、虚拟机、容器中的至少一种。进一步地，上述计算实例可以是一台或者多台。例如，处理模块1302可以包括运行在多个主机/虚拟机/容器上的代码。需要说明的是，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的区域(region)中，也可以分布在不同的region中。进一步地，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的可用区(availability zone，AZ)中，也可以分布在不同的AZ中，每个AZ包括一个数据中心或多个地理位置相近的数据中心。其中，通常一个region可以包括多个AZ。Module As an example of a software functional unit, the processing module 1302 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above computing instance may be one or more. For example, processing module 1302 may include code running on multiple hosts/virtual machines/containers. It should be noted that multiple hosts/virtual machines/containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, multiple hosts/virtual machines/containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs. Each AZ includes one data center or multiple AZs. geographically close data centers. Among them, usually a region can include multiple AZs.

同样，用于运行该代码的多个主机/虚拟机/容器可以分布在同一个虚拟私有云(virtual private cloud，VPC)中，也可以分布在多个VPC中。其中，通常一个VPC设置在一个region内，同一region内两个VPC之间，以及不同region的VPC之间跨区通信需在每个VPC内设置通信网关，经通信网关实现VPC之间的互连。Likewise, the multiple hosts/VMs/containers used to run the code can be distributed in the same virtual private cloud (VPC), or across multiple VPCs. Among them, usually a VPC is set up in a region. Cross-region communication between two VPCs in the same region and between VPCs in different regions requires a communication gateway in each VPC, and the interconnection between VPCs is realized through the communication gateway. .

模块作为硬件功能单元的一种举例，处理模块1302可以包括至少一个计算设备，如服务器等。或者，处理模块1302也可以是利用专用集成电路(application-specific integrated circuit，ASIC)实现、或可编程逻辑器件(programmable logic device，PLD)实现的设备等。其中，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)、现场可编程门阵列(field-programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合实现。Module As an example of a hardware functional unit, the processing module 1302 may include at least one computing device, such as a server. Alternatively, the processing module 1302 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above-mentioned PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL), or any combination thereof.

处理模块1302包括的多个计算设备可以分布在相同的region中，也可以分布在不同的region中。处理模块1302包括的多个计算设备可以分布在相同的AZ中，也可以分布在不同的AZ中。同样，处理模块1302包括的多个计算设备可以分布在同一个VPC中，也可以分布在多个VPC中。其中，所述多个计算设备可以是服务器、ASIC、PLD、CPLD、FPGA和GAL等计算设备的任意组合。Multiple computing devices included in the processing module 1302 may be distributed in the same region or in different regions. Multiple computing devices included in the processing module 1302 may be distributed in the same AZ or in different AZs. Similarly, multiple computing devices included in the processing module 1302 may be distributed in the same VPC or in multiple VPCs. The plurality of computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

需要说明的是，在其他实施例中，处理模块1302可以用于执行图12中所示的方法中的任意步骤，图13中所示的处理模块1302可以用于执行上述图12中提供的方法中的任意步骤。图13中所示的获取模块1301和处理模块1302负责实现的步骤可根据需要指定，通过获取模块1301和处理模块1302分别实现上述图12中提供的方法中不同的步骤来实现图像处理装置1300的全部功能。It should be noted that in other embodiments, the processing module 1302 can be used to perform any steps in the method shown in Figure 12, and the processing module 1302 shown in Figure 13 can be used to perform the method provided in Figure 12. any step in . The steps that the acquisition module 1301 and the processing module 1302 shown in Figure 13 are responsible for implementing can be specified as needed. The acquisition module 1301 and the processing module 1302 respectively implement different steps in the method provided in Figure 12 to implement the image processing device 1300. Full functionality.

本申请还提供一种计算设备1400。如图14所示，计算设备1400包括：总线1402、处理器1404、存储器1406和通信接口1408。处理器1404、存储器1406和通信接口1408之间通过总线1402通信。计算设备1400可以是服务器或终端设备。应理解，本申请不限定计算设备1400中的处理器、存储器的个数。The present application also provides a computing device 1400. As shown in Figure 14, computing device 1400 includes: bus 1402, processor 1404, memory 1406, and communication interface 1408. The processor 1404, the memory 1406 and the communication interface 1408 communicate through a bus 1402. Computing device 1400 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1400.

总线1402可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图14中仅用一条线表示，但并不表示仅有一根总线或一种类型的总线。总线1404可包括在计算设备1400各个部件(例如，存储器1406、处理器1404、通信接口1408)之间传送信息的通路。The bus 1402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one line is used in Figure 14, but it does not mean that there is only one bus or one type of bus. Bus 1404 may include a path that carries information between various components of computing device 1400 (eg, memory 1406, processor 1404, communications interface 1408).

处理器1404可以包括中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。The processor 1404 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). any one or more of them.

存储器1406可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。处理器144还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)。Memory 1406 may include volatile memory, such as random access memory (RAM). The processor 144 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid state drive (solid state drive). drive, SSD).

存储器1406中存储有可执行的程序代码，处理器1404执行该可执行的程序代码以分别实现前述图13中所示的获取模块1301和处理模块1302的功能，从而实现上述实施例中方法的全部或部分步骤。也即，存储器146上存有用于执行上述实施例方法中全部或部分步骤的指令。The memory 1406 stores executable program code, and the processor 1404 executes the executable program code to respectively realize the functions of the acquisition module 1301 and the processing module 1302 shown in Figure 13, thereby realizing all the methods in the above embodiments. or partial steps. That is, the memory 146 stores instructions for executing all or part of the steps in the methods of the above embodiments.

或者，存储器1406中存储有可执行的代码，处理器1404执行该可执行的代码以分别实现前述图像处理装置1300的功能，从而实现上述实施例方法中全部或部分步骤。也即，存储器1406上存有用于执行上述实施例方法中全部或部分步骤的指令。Alternatively, executable code is stored in the memory 1406, and the processor 1404 executes the executable code to respectively implement the functions of the aforementioned image processing device 1300, thereby implementing all or part of the steps in the method of the above embodiment. That is, the memory 1406 stores instructions for executing all or part of the steps in the methods of the above embodiments.

通信接口1403使用例如但不限于网络接口卡、收发器一类的收发模块，来实现计算设备140与其他设备或通信网络之间的通信。The communication interface 1403 uses transceiver modules such as but not limited to network interface cards and transceivers to implement the communication between the computing device 140 and other Communication between devices or communication networks.

本申请实施例还提供了一种计算设备集群。该计算设备集群包括至少一台计算设备。该计算设备可以是服务器，例如是中心服务器、边缘服务器，或者是本地数据中心中的本地服务器。在一些实施例中，计算设备也可以是台式机、笔记本电脑或者智能手机等终端设备。An embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

如图15所示，所述计算设备集群包括至少一个计算设备1400。计算设备集群中的一个或多个计算设备1400中的存储器1406中可以存有相同的用于执行上述实施例方法中全部或部分步骤的指令。As shown in Figure 15, the computing device cluster includes at least one computing device 1400. The memory 1406 in one or more computing devices 1400 in the computing device cluster may store the same instructions for executing all or part of the steps in the above embodiment method.

在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备1400的存储器1406中也可以分别存有用于执行上述实施例方法中全部或部分步骤的部分指令。换言之，一个或多个计算设备1400的组合可以共同执行用于执行上述实施例方法中全部或部分步骤的指令。In some possible implementations, the memory 1406 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for executing all or part of the steps in the above embodiment methods. In other words, a combination of one or more computing devices 1400 may jointly execute instructions for performing all or part of the steps in the above embodiment methods.

需要说明的是，计算设备集群中的不同的计算设备1400中的存储器1406可以存储不同的指令，分别用于执行图像处理装置1300的部分功能。也即，不同的计算设备1400中的存储器1406存储的指令可以实现前述图13中所示的获取模块814和处理模块820中的一个或多个模块的功能。It should be noted that the memory 1406 in different computing devices 1400 in the computing device cluster can store different instructions, respectively used to execute part of the functions of the image processing apparatus 1300. That is, the instructions stored in the memory 1406 in different computing devices 1400 can implement the functions of one or more modules in the acquisition module 814 and the processing module 820 shown in FIG. 13 .

在一些可能的实现方式中，计算设备集群中的一个或多个计算设备可以通过网络连接。其中，所述网络可以是广域网或局域网等等。In some possible implementations, one or more computing devices in a cluster of computing devices may be connected through a network. Wherein, the network may be a wide area network or a local area network, etc.

基于上述实施例中的方法，本申请实施例提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，当计算机程序在处理器上运行时，使得处理器执行上述实施例中的方法。Based on the methods in the above embodiments, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is run on a processor, it causes the processor to execute the steps in the above embodiments. Methods.

基于上述实施例中的方法，本申请实施例提供了一种计算机程序产品，当计算机程序产品在处理器上运行时，使得处理器执行上述实施例中的方法。Based on the methods in the above embodiments, embodiments of the present application provide a computer program product, which when the computer program product is run on a processor, causes the processor to execute the methods in the above embodiments.

可以理解的是，本申请的实施例中的处理器可以是中央处理单元(central processing unit，CPU)，还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、晶体管逻辑器件，硬件部件或者其任意组合。通用处理器可以是微处理器，也可以是任何常规的处理器。It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), or other general-purpose processor, digital signal processor (DSP), or application-specific integrated circuit (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

本申请的实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable rom，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. Software instructions can be composed of corresponding software modules, which can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable rom) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or other well-known in the art any other form of storage media. An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage media may be located in an ASIC.

在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者通过所述计算机可读存储介质进行传输。所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(solid state disk，SSD))等。In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server or data center to another website through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. , computer, server or data center for transmission. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state disk (SSD)), etc.

可以理解的是，在本申请的实施例中涉及的各种数字编号仅为描述方便进行的区分，并不用来限制本申请的实施例的范围。 It can be understood that the various numerical numbers involved in the embodiments of the present application are only for convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

一种图像处理方法，其特征在于，所述方法包括：An image processing method, characterized in that the method includes:

获取第一图像；Get the first image;

对于所述第一图像中的任意一个像素点，利用N个(N≥2)与所述任意一个像素点关联的查找表，确定所述任意一个像素点所对应的图像块，所述图像块中包括单个或多个像素点，其中，每个所述查找表的索引区域均包含所述任意一个像素点，且不同的所述查找表的索引区域不同且至少有部分相同；For any pixel point in the first image, use N (N≥2) lookup tables associated with the any pixel point to determine the image block corresponding to the any pixel point, and the image block includes single or multiple pixels, wherein the index area of each look-up table contains any one of the pixels, and the index areas of different look-up tables are different and at least partially the same;

根据所述第一图像中各个像素点所对应的图像块，得到第二图像，其中，所述第二图像的分辨率高于或等于所述第一图像的分辨率；Obtain a second image based on the image blocks corresponding to each pixel point in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image;

输出所述第二图像。Output the second image.
根据权利要求1所述的方法，其特征在于，所述利用N个(N≥2)与所述任意一个像素点关联的查找表，确定所述任意一个像素点所对应的图像块，具体包括：The method according to claim 1, characterized in that the image block corresponding to any one pixel is determined by using N (N≥2) lookup tables associated with any one pixel, specifically including: :

将所述N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，每个所述第一子图像块中均包括单个或多个像素点；Input the index area of each lookup table in the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks. Each of the first sub-image blocks includes single or multiple pixels. ;

将所述N个第一子图像块中相同位置的像素点均相加并求平均，以得到所述任意一个像素点所对应的图像块。The pixel points at the same position in the N first sub-image blocks are added and averaged to obtain the image block corresponding to any one of the pixel points.
根据权利要求2所述的方法，其特征在于，所述将所述N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，具体包括：The method according to claim 2, characterized in that said inputting the index area of each lookup table in the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks specifically includes:

针对所述N个查找表中任意一个查找表的索引区域，将所述任意一个查找表的索引区域以所述任意一个像素点为中心，并沿第一方向旋转M次，以得到M个索引区域，M≥1；For the index area of any one of the N look-up tables, center the index area of any one of the look-up tables on any one pixel and rotate it M times along the first direction to obtain M indexes. Area, M≥1;

基于所述M个索引区域中每个索引区域对应的查找表，确定所述M个索引区域各自所对应的第二子图像块，以得到M个第二子图像块，每个所述第二子图像块中均包括单个或多个像素点；Based on the lookup table corresponding to each index area in the M index areas, the second sub-image blocks corresponding to each of the M index areas are determined to obtain M second sub-image blocks, each of the second sub-image blocks Each sub-image block includes single or multiple pixels;

基于所述M个索引区域各自所对应的旋转角度，分别对所述M个索引区域各自所对应的第二子图像块沿与所述第一方向相反的方向旋转相同的角度，以得到M个第三子图像块；Based on the rotation angles corresponding to each of the M index areas, the second sub-image blocks corresponding to each of the M index areas are rotated by the same angle in the direction opposite to the first direction to obtain M The third sub-image block;

将所述任意一个查找表的索引区域所对应的第一子图像块和所述M个第三子图像块中相同位置的像素点均相加并求平均，以及，将计算结果作为所述任意一个查找表的索引区域所对应的第一子图像块。Add and average the pixel points at the same position in the first sub-image block corresponding to the index area of any one of the lookup tables and the M third sub-image blocks, and use the calculation result as the arbitrary The first sub-image block corresponding to the index area of a lookup table.
根据权利要求1所述的方法，其特征在于，所述N个查找表中第i个查找表的输出为第(i+1)个查找表的输入，所述N个查找表中第1个查找表的输入为第1个查找表的索引区域，所述N个查找表中第N个查找表的输出为所述任意一个像素点所对应的图像块。The method according to claim 1, characterized in that the output of the i-th lookup table among the N lookup tables is the input of the (i+1)th lookup table, and the 1st among the N lookup tables The input of the lookup table is the index area of the first lookup table, and the output of the Nth lookup table among the N lookup tables is the image block corresponding to any one of the pixels.
根据权利要求1所述的方法，其特征在于，所述N个查找表由M个查找表集合中所包含的查找表构成，每个所述查找表集合中均包括多个查找表；The method according to claim 1, characterized in that the N look-up tables are composed of look-up tables included in M look-up table sets, and each of the look-up table sets includes multiple look-up tables;

其中，第j个查找表集合中各个查找表的输入分别为以第(j-1)个查找表集合的输出为基准确定出的所述第j个查找表集合中相应的查找表的索引区域；Wherein, the input of each lookup table in the jth lookup table set is respectively the index area of the corresponding lookup table in the jth lookup table set determined based on the output of the (j-1)th lookup table set. ;

第1个查找表集合中各个查找表的输入分别为以所述任意一个像素点为基准确定出的所述第1个查找表集合中相应的查找表的索引区域；The input of each lookup table in the first lookup table set is the index area of the corresponding lookup table in the first lookup table set determined based on any one pixel point;

第M个查找表集合的输出为所述任意一个像素点所对应的图像块。The output of the M-th lookup table set is the image block corresponding to any one of the pixels.
一种图像处理装置，其特征在于，所述装置包括：An image processing device, characterized in that the device includes:

获取模块，用于获取第一图像；Acquisition module, used to acquire the first image;

处理模块，用于对于所述第一图像中的任意一个像素点，利用N个(N≥2)与所述任意一个像素点关联的查找表，确定所述任意一个像素点所对应的图像块，所述图像块中包括单个或多个像素点，其中，每个所述查找表的索引区域均包含所述任意一个像素点，且不同的所述查找表的索引区域不同且至少有部分相同；A processing module configured to use N (N≥2) lookup tables associated with any pixel point in the first image to determine the image block corresponding to the any pixel point. , the image block includes single or multiple pixels, wherein the index area of each look-up table contains any one of the pixels, and the index areas of different look-up tables are different and at least partially the same. ;

所述处理模块，还用于根据所述第一图像中各个像素点所对应的图像块，得到第二图像，其中，所述第二图像的分辨率高于或等于所述第一图像的分辨率；以及，输出所述第二图像。The processing module is further configured to obtain a second image based on the image blocks corresponding to each pixel point in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image. rate; and, output the second image.
根据权利要求6所述的装置，其特征在于，所述处理模块在利用N个(N≥2)与所述任意一个像素点关联的查找表，确定所述任意一个像素点所对应的图像块时，具体用于：The device according to claim 6, wherein the processing module uses N (N≥2) lookup tables associated with any one of the pixels to determine the image block corresponding to the any one of the pixels. When, it is specifically used for:

将所述N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块，每个所述第一子图像块中均包括单个或多个像素点；Input the index area of each lookup table in the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks. Each of the first sub-image blocks includes single or multiple pixels. ;

将所述N个第一子图像块中相同位置的像素点均相加并求平均，以得到所述任意一个像素点所对应的图像块。Add the pixels at the same position in the N first sub-image blocks and average them to obtain the pixel corresponding to any one pixel. Image block.
根据权利要求7所述的装置，其特征在于，所述处理模块在将所述N个查找表中每个查找表的索引区域输入至相应的查找表中，得到N个第一子图像块时，具体用于：The device according to claim 7, wherein the processing module inputs the index area of each lookup table in the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks. , specifically used for:

针对所述N个查找表中任意一个查找表的索引区域，将所述任意一个查找表的索引区域以所述任意一个像素点为中心，并沿第一方向旋转M次，以得到M个索引区域，M≥1；For the index area of any one of the N look-up tables, center the index area of any one of the look-up tables on any one pixel and rotate it M times along the first direction to obtain M indexes. Area, M≥1;

基于所述M个索引区域中每个索引区域对应的查找表，确定所述M个索引区域各自所对应的第二子图像块，以得到M个第二子图像块，每个所述第二子图像块中均包括单个或多个像素点；Based on the lookup table corresponding to each index area in the M index areas, the second sub-image blocks corresponding to each of the M index areas are determined to obtain M second sub-image blocks, each of the second sub-image blocks Each sub-image block includes single or multiple pixels;

基于所述M个索引区域各自所对应的旋转角度，分别对所述M个索引区域各自所对应的第二子图像块沿与所述第一方向相反的方向旋转相同的角度，以得到M个第三子图像块；Based on the rotation angles corresponding to each of the M index areas, the second sub-image blocks corresponding to each of the M index areas are rotated by the same angle in the direction opposite to the first direction to obtain M The third sub-image block;

将所述任意一个查找表的索引区域所对应的第一子图像块和所述M个第三子图像块中相同位置的像素点均相加并求平均，以及，将计算结果作为所述任意一个查找表的索引区域所对应的第一子图像块。Add and average the pixel points at the same position in the first sub-image block corresponding to the index area of any one of the lookup tables and the M third sub-image blocks, and use the calculation result as the arbitrary The first sub-image block corresponding to the index area of a lookup table.
根据权利要求6所述的装置，其特征在于，所述N个查找表中第i个查找表的输出为第(i+1)个查找表的输入，所述N个查找表中第1个查找表的输入为第1个查找表的索引区域，所述N个查找表中第N个查找表的输出为所述任意一个像素点所对应的图像块。The device according to claim 6, characterized in that the output of the i-th lookup table among the N lookup tables is the input of the (i+1)th lookup table, and the 1st among the N lookup tables The input of the lookup table is the index area of the first lookup table, and the output of the Nth lookup table among the N lookup tables is the image block corresponding to any one of the pixels.
根据权利要求6所述的方法，其特征在于，所述N个查找表由M个查找表集合中所包含的查找表构成，每个所述查找表集合中均包括多个查找表；The method according to claim 6, characterized in that the N look-up tables are composed of look-up tables included in M look-up table sets, and each of the look-up table sets includes multiple look-up tables;

其中，第j个查找表集合中各个查找表的输入分别为以第(j-1)个查找表集合的输出为基准确定出的所述第j个查找表集合中相应的查找表的索引区域；Wherein, the input of each lookup table in the jth lookup table set is respectively the index area of the corresponding lookup table in the jth lookup table set determined based on the output of the (j-1)th lookup table set. ;

第1个查找表集合中各个查找表的输入分别为以所述任意一个像素点为基准确定出的所述第1个查找表集合中相应的查找表的索引区域；The input of each lookup table in the first lookup table set is the index area of the corresponding lookup table in the first lookup table set determined based on any one pixel point;

第M个查找表集合的输出为所述任意一个像素点所对应的图像块。The output of the M-th lookup table set is the image block corresponding to any one of the pixels.
一种计算设备，其特征在于，包括：A computing device, characterized by including:

至少一个存储器，用于存储程序；At least one memory for storing programs;

至少一个处理器，用于执行所述存储器存储的程序；At least one processor for executing programs stored in the memory;

其中，当所述存储器存储的程序被执行时，所述处理器用于执行如权利要求1-5任一所述的方法。Wherein, when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1-5.
一种计算设备集群，其特征在于，包括至少一个计算设备，每个计算设备均包括处理器和存储器；A computing device cluster, characterized by including at least one computing device, each computing device including a processor and a memory;

所述至少一个计算设备的处理器用于执行所述至少一个计算设备的存储器中存储的指令，以使得所述计算设备集群执行如权利要求1-5任一所述的方法。The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the cluster of computing devices performs the method according to any one of claims 1-5.
一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，当所述计算机程序在处理器上运行时，使得所述处理器执行如权利要求1-5任一所述的方法。A computer-readable storage medium that stores a computer program. When the computer program is run on a processor, it causes the processor to execute the method as described in any one of claims 1-5. .
一种计算机程序产品，其特征在于，当所述计算机程序产品在处理器上运行时，使得所述处理器执行如权利要求1-5任一所述的方法。 A computer program product, characterized in that, when the computer program product is run on a processor, it causes the processor to execute the method according to any one of claims 1-5.