WO2024041027A1

WO2024041027A1 - Panoramic-image processing method, and computer device and storage medium

Info

Publication number: WO2024041027A1
Application number: PCT/CN2023/091779
Authority: WO
Inventors: 洪国伟; 梁乔惠; 董治; 姜涛
Original assignee: 腾讯音乐娱乐科技（深圳）有限公司
Priority date: 2022-08-26
Filing date: 2023-04-28
Publication date: 2024-02-29
Also published as: CN115358949A

Abstract

The present application relates to a panoramic-image processing method, and a computer device, a storage medium and a computer program product. The method comprises: performing deformable convolution processing on a panoramic image to be processed, so as to obtain a feature image of said panoramic image (S101); performing regression processing on the feature image, so as to obtain image transformation information of said panoramic image (S102); according to the image transformation information and corrected pixel positions of said panoramic image, obtaining mapped pixel positions, which are in the panoramic image and correspond to the corrected pixel positions, wherein the corrected pixel positions are represented as coordinate positions of pixels in a corrected image corresponding to said panoramic image (S103); and performing pixel correction on said panoramic image according to pixel information of the mapped pixel positions, so as to obtain the corrected image corresponding to said panoramic image (S104).

Description

全景图像处理方法、计算机设备和存储介质Panoramic image processing method, computer equipment and storage medium

相关申请的交叉引用Cross-references to related applications

本申请要求于2022年8月26日提交中国专利局、申请号为202211030788.1、发明名称为“全景图像处理方法、计算机设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority to the Chinese patent application filed with the China Patent Office on August 26, 2022, with application number 202211030788.1 and the invention title "Panorama Image Processing Method, Computer Equipment and Storage Medium", the entire content of which is incorporated by reference. in this application.

技术领域Technical field

本申请涉及图像处理技术领域，特别是涉及一种全景图像处理方法、计算机设备、存储介质和计算机程序产品。The present application relates to the field of image processing technology, and in particular to a panoramic image processing method, computer equipment, storage media and computer program products.

背景技术Background technique

全景相机能够拍摄比传统镜头更广的视野，全景相机可以拍摄到传统镜头拍摄不到的侧面、上面和背面等多视角画面。Panoramic cameras can capture a wider field of view than traditional lenses. Panoramic cameras can capture multi-angle images from the side, top, and back that traditional lenses cannot capture.

然而，广泛的视角也导致全景图像的成像存在一些扭曲和畸变，畸变成像会导致普通网络处理全景图像或全景视频的效果较差。However, the wide viewing angle also leads to some distortion and distortion in the imaging of panoramic images. Distorted imaging will lead to poor performance of ordinary networks in processing panoramic images or panoramic videos.

发明内容Contents of the invention

根据本申请的各种实施例，提供一种全景图像处理方法、计算机设备、计算机可读存储介质和计算机程序产品，以至少解决相关技术中网络对全景图像的处理效果较差的问题。本公开的技术方案如下：According to various embodiments of the present application, a panoramic image processing method, a computer device, a computer-readable storage medium and a computer program product are provided to at least solve the problem in the related art that the network has poor processing effect on panoramic images. The technical solutions of the present disclosure are as follows:

第一方面，本申请提供了一种全景图像处理方法。所述方法包括：In a first aspect, this application provides a panoramic image processing method. The methods include:

对待处理的全景图像进行可变形卷积处理，得到所述全景图像的特征图像；Perform deformable convolution processing on the panoramic image to be processed to obtain a characteristic image of the panoramic image;

对所述特征图像进行回归处理，得到所述全景图像的图像变换信息；Perform regression processing on the feature image to obtain image transformation information of the panoramic image;

根据所述图像变换信息和所述全景图像的矫正像素位置，得到所述全景图像中与所述矫正像素位置对应的映射像素位置；所述矫正像素位置表示为所述全景图像对应的矫正图像中像素的坐标位置；According to the image transformation information and the corrected pixel position of the panoramic image, the mapped pixel position corresponding to the corrected pixel position in the panoramic image is obtained; the corrected pixel position is expressed as the corrected pixel position corresponding to the panoramic image. The coordinate position of the pixel;

根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像。According to the pixel information of the mapped pixel position, pixel correction is performed on the panoramic image to obtain a corrected image corresponding to the panoramic image.

在其中一个实施例中，对待处理的全景图像进行可变形卷积处理，得到所述全景图像的特征图像，包括：In one embodiment, the panoramic image to be processed is subjected to deformable convolution processing to obtain the characteristic image of the panoramic image, including:

根据所述全景图像的采样位置偏移量，对所述全景图像的采样位置进行偏移处理，得到所述全景图像的偏移后采样位置；According to the sampling position offset of the panoramic image, perform offset processing on the sampling position of the panoramic image to obtain the offset sampling position of the panoramic image;

对所述偏移后采样位置进行线性插值处理，得到所述全景图像对应的偏移图像；Perform linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image;

对所述偏移图像进行卷积处理，得到所述全景图像的特征图像。Convolution processing is performed on the offset image to obtain a characteristic image of the panoramic image.

在其中一个实施例中，在根据所述全景图像的采样位置偏移量，对所述全景图像的采样位置进行偏移处理，得到所述全景图像的偏移后采样位置之前，还包括：In one embodiment, before performing offset processing on the sampling position of the panoramic image according to the sampling position offset of the panoramic image to obtain the offset sampling position of the panoramic image, the method further includes:

对所述全景图像进行卷积处理，得到所述全景图像的采样位置偏移量；所述采样位置偏移量的数量与所述卷积处理中的卷积核的数量相等。Perform convolution processing on the panoramic image to obtain a sampling position offset of the panoramic image; the number of sampling position offsets is equal to the number of convolution kernels in the convolution process.

在其中一个实施例中，对所述特征图像进行回归处理，得到所述全景图像的图像变换信息，包括：In one embodiment, regression processing is performed on the feature image to obtain image transformation information of the panoramic image, including:

通过回归网络对所述特征图像进行仿射变换处理，得到所述全景图像的图像变换信息；所述回归网络是依据样本全景图像对待训练的回归网络进行训练得到。The feature image is subjected to affine transformation processing through a regression network to obtain the image transformation information of the panoramic image; the regression network is obtained by training the regression network to be trained based on the sample panoramic image.

在其中一个实施例中，根据所述图像变换信息和所述全景图像的矫正像素位置，得到所述全景图像中与所述矫正像素位置对应的映射像素位置，包括：In one embodiment, obtaining the mapped pixel position corresponding to the corrected pixel position in the panoramic image according to the image transformation information and the corrected pixel position of the panoramic image includes:

将所述图像变换信息和所述矫正像素位置进行相乘，得到所述全景图像中与所述矫正像素位置对应的映射像素位置。 The image transformation information and the corrected pixel position are multiplied to obtain a mapped pixel position corresponding to the corrected pixel position in the panoramic image.

在其中一个实施例中，根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像，包括：In one embodiment, pixel correction is performed on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image, including:

在所述映射像素位置的坐标值不是整数的情况下，对所述映射像素位置的坐标值进行线性插值处理，得到目标像素位置；所述目标像素位置的坐标值为整数；When the coordinate value of the mapped pixel position is not an integer, perform linear interpolation processing on the coordinate value of the mapped pixel position to obtain the target pixel position; the coordinate value of the target pixel position is an integer;

根据所述目标像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像。According to the pixel information of the target pixel position, pixel correction is performed on the panoramic image to obtain a corrected image corresponding to the panoramic image.

在所述映射像素位置的坐标值是整数的情况下，将所述矫正像素位置的像素信息，更新为所述矫正像素位置对应的映射像素位置的像素信息，得到所述全景图像对应的矫正图像。When the coordinate value of the mapped pixel position is an integer, the pixel information of the corrected pixel position is updated to the pixel information of the mapped pixel position corresponding to the corrected pixel position, and a corrected image corresponding to the panoramic image is obtained. .

在其中一个实施例中，所述方法还包括：In one embodiment, the method further includes:

对所述矫正图像进行图像处理，得到所述矫正图像对应的目标全景图像。Perform image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image.

在其中一个实施例中，对所述矫正图像进行图像处理，得到所述矫正图像对应的目标全景图像，包括：In one embodiment, image processing is performed on the corrected image to obtain a target panoramic image corresponding to the corrected image, including:

对所述矫正图像进行超分处理，得到所述矫正图像对应的目标超分辨率全景图像；所述目标超分辨率全景图像的图像分辨率高于基于所述待处理的全景图像得到的超分辨率图像；Perform super-resolution processing on the corrected image to obtain a target super-resolution panoramic image corresponding to the corrected image; the image resolution of the target super-resolution panoramic image is higher than the super-resolution obtained based on the panoramic image to be processed. rate image;

或者，or,

对所述矫正图像进行去噪处理，得到所述矫正图像对应的目标去噪全景图像；所述目标去噪全景图像包含的图像噪声少于基于所述待处理的全景图像得到的去噪后全景图像。Perform denoising processing on the corrected image to obtain a target denoised panoramic image corresponding to the corrected image; the target denoised panoramic image contains less image noise than the denoised panorama obtained based on the panoramic image to be processed. image.

第二方面，本申请还提供了一种计算机设备。所述计算机设备包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现以下步骤：In a second aspect, this application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

第三方面，本申请还提供了一种计算机可读存储介质。所述计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现以下步骤：In a third aspect, this application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by the processor, the following steps are implemented:

对待处理全景图像进行可变形卷积处理，得到所述全景图像的特征图像；Perform deformable convolution processing on the panoramic image to be processed to obtain the characteristic image of the panoramic image;

第四方面，本申请还提供了一种计算机程序产品。所述计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现以下步骤：In a fourth aspect, this application also provides a computer program product. The computer program product includes a computer program that implements the following steps when executed by a processor:

本发明的一个或多个实施例的细节在下面的附图和描述中提出。本发明的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。 The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects and advantages of the invention will become apparent from the description, drawings and claims.

附图说明Description of drawings

为了更好地描述和说明这里公开的那些发明的实施例和或示例，可以参考一幅或多幅附图。用于描述附图的附加细书或示例不应当被认为是对所公开的发明、目前描述的实施例和或示例以及目前理解的这些发明的最佳模式中的任何一者的范围的限制。To better describe and illustrate embodiments and/or examples of those inventions disclosed herein, reference may be made to one or more of the accompanying drawings. The additional details or examples used to describe the drawings should not be construed as limiting the scope of any of the disclosed inventions, the embodiments and/or examples presently described, and the best modes currently understood of these inventions.

图1为一个实施例中全景图像处理方法的流程示意图；Figure 1 is a schematic flowchart of a panoramic image processing method in one embodiment;

图2为一个实施例中全景图像处理方法的结构环境图；Figure 2 is a structural environment diagram of a panoramic image processing method in one embodiment;

图3为一个实施例中得到全景图像的特征图像步骤的流程示意图；Figure 3 is a schematic flowchart of the steps of obtaining the characteristic image of the panoramic image in one embodiment;

图4为一个实施例中全景图像处理方法的流程示意图；Figure 4 is a schematic flowchart of a panoramic image processing method in one embodiment;

图5为另一个实施例中全景图像处理方法的结构示意图；Figure 5 is a schematic structural diagram of a panoramic image processing method in another embodiment;

图6为一个实施例中全景图像处理方法的流程环境图；Figure 6 is a process environment diagram of a panoramic image processing method in one embodiment;

图7为一个实施例中计算机设备的内部结构图。Figure 7 is an internal structure diagram of a computer device in one embodiment.

具体实施方式Detailed ways

为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

在一个实施例中，如图1所示，提供了一种全景图像处理方法，本实施例以该方法应用于服务器进行举例说明，可以理解的是，该方法也可以应用于终端，还可以应用于包括终端和服务器的***，并通过终端和服务器的交互实现。本实施例中，该方法包括以下步骤：In one embodiment, as shown in Figure 1, a panoramic image processing method is provided. This embodiment illustrates the application of this method to a server. It can be understood that this method can also be applied to terminals, and can also be applied to It is based on a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

步骤S101，对待处理的全景图像进行可变形卷积处理，得到全景图像的特征图像。Step S101: Perform deformable convolution processing on the panoramic image to be processed to obtain a characteristic image of the panoramic image.

步骤S102，对特征图像进行回归处理，得到全景图像的图像变换信息；Step S102, perform regression processing on the feature image to obtain image transformation information of the panoramic image;

其中，待处理的全景图像是指需要进行图像处理的全景图像。待处理的全景图像可以是预先存储在服务器中的，也可以是终端发送到服务器中的，当然还可以是通过其他方式得到的。其中，全景图像是指360度球形范围内景致的图像。需要说明的是，在本方法的各步骤中，除了可以使用全景图像，还可以使用全景视频。Wherein, the panoramic image to be processed refers to the panoramic image that requires image processing. The panoramic image to be processed may be stored in the server in advance, or may be sent to the server by the terminal. Of course, it may also be obtained through other methods. Among them, panoramic images refer to images of scenery within a 360-degree spherical range. It should be noted that in each step of this method, in addition to using panoramic images, panoramic videos can also be used.

其中，图像变换信息指的是全景图像进行图像旋转、缩放和平移等线性变换的信息。图像变换信息包括但不限于是全景图像的仿射变换信息和投影变换信息。Among them, the image transformation information refers to the information of linear transformation such as image rotation, scaling and translation of the panoramic image. Image transformation information includes, but is not limited to, affine transformation information and projective transformation information of the panoramic image.

图2为上述全景图像处理方法的结构示意图，如图2所示，上述步骤S101和步骤S102中，通过可变形空间变换模型中的可变形定位网络(Deformable localisation net)对该全景图像进行处理，得到全景图像的图像变换信息。具体地，服务器获取待处理的全景图像，对该全景图像进行卷积处理，得到该全景图像对应的采样位置偏移量；先将采样位置偏移量加在全景图像的像素位置上，再使用卷积核对叠加后的全景图像进行多次卷积处理，得到全景图像的特征图像。服务器将该特征图像输入到回归网络进行回归处理；通过回归网络对特征图像进行变换，得到全景图像的图像变换信息(例如图像变换矩阵)。其中，图像变换矩阵的维度是依据回归网络选择的针对特征图像的变换类型确定的。其中，变换类型可以但不限于是仿射变换和投影变换。例如，回归网络对特征图像进行仿射变换，则可以得到一个维度为2*3的图像变换矩阵。Figure 2 is a schematic structural diagram of the above panoramic image processing method. As shown in Figure 2, in the above steps S101 and step S102, the panoramic image is processed through the deformable localization network (Deformable localization net) in the deformable space transformation model. Obtain the image transformation information of the panoramic image. Specifically, the server obtains the panoramic image to be processed, performs convolution processing on the panoramic image, and obtains the sampling position offset corresponding to the panoramic image; first adds the sampling position offset to the pixel position of the panoramic image, and then uses The convolution kernel performs multiple convolution processes on the superimposed panoramic image to obtain the characteristic image of the panoramic image. The server inputs the feature image into the regression network for regression processing; the feature image is transformed through the regression network to obtain the image transformation information (such as an image transformation matrix) of the panoramic image. Among them, the dimension of the image transformation matrix is determined based on the transformation type selected by the regression network for the feature image. Among them, the transformation type can be, but is not limited to, affine transformation and projective transformation. For example, if the regression network performs affine transformation on the feature image, an image transformation matrix with a dimension of 2*3 can be obtained.

在一实施方式中，图2中的可变形定位网络可以采用全卷积的网络结构，还可以采用全连接的网络结构，也可以是卷积和连接两者结合的网络结构，当然更可以根据全景图像的后续图像处理方式对可变形定位网络中的网络结构进行调整。In one embodiment, the deformable positioning network in Figure 2 can adopt a fully convolutional network structure, a fully connected network structure, or a network structure that combines convolution and connection. Of course, it can also be based on The subsequent image processing of panoramic images adjusts the network structure in the deformable positioning network.

步骤S103，根据图像变换信息和全景图像的矫正像素位置，得到全景图像中与矫正像素位置对应的映射像素位置；矫正像素位置表示为全景图像对应的矫正图像中像素的坐标位置。Step S103, obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image based on the image transformation information and the corrected pixel position of the panoramic image; the corrected pixel position is expressed as the coordinate position of the pixel in the corrected image corresponding to the panoramic image.

其中，全景图像的像素位置是指全景图像中各个像素的坐标。Among them, the pixel position of the panoramic image refers to the coordinates of each pixel in the panoramic image.

其中，矫正像素位置指的是全景图像对应的矫正图像中各个像素的坐标。矫正像素位置是依据预先设置的全景图像对应的矫正图像的尺寸确定的。Among them, the corrected pixel position refers to the coordinates of each pixel in the corrected image corresponding to the panoramic image. The corrected pixel position is determined based on the size of the corrected image corresponding to the preset panoramic image.

如图2所示，在步骤S103中，通过可变形空间变换模型中的网络生成器(Grid generator)来得到全景图像中与矫正像素位置对应的映射像素位置。具体地，服务器在获取矫正图像的尺寸后，可以确定矫正图像中的矫正像素位置，进而根据图像变换矩阵，来确定全景图像中与矫正像素位置对应的映射像素位置，即得到映射像素位置与矫正像素位置之间的映射关系，以便后续步骤将映射像素位置的像素信息矫正到矫正像素位置。例如，矫正像素位置为(2，4)，服务器根据图像变换矩阵，得到矫正像素位置(2，4)在全景图像中对应的映射像素位置为(5，6)，可建立映射像素位置(5，6)与矫正像素位置为(2，4)之间的映射关系。As shown in Figure 2, in step S103, the network generator (Grid generator) in the deformable space transformation model is used to obtain The mapped pixel position in the panoramic image corresponding to the rectified pixel position. Specifically, after obtaining the size of the corrected image, the server can determine the corrected pixel position in the corrected image, and then determine the mapped pixel position corresponding to the corrected pixel position in the panoramic image according to the image transformation matrix, that is, the mapped pixel position and the corrected pixel position are obtained. The mapping relationship between pixel positions is used to correct the pixel information of the mapped pixel positions to the corrected pixel positions in subsequent steps. For example, the corrected pixel position is (2, 4). The server obtains the corrected pixel position (2, 4) according to the image transformation matrix. The corresponding mapped pixel position in the panoramic image is (5, 6). The mapped pixel position (5 , 6) and the mapping relationship between the corrected pixel position (2, 4).

步骤S104，根据映射像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像。Step S104: Perform pixel correction on the panoramic image according to the pixel information of the mapped pixel positions to obtain a corrected image corresponding to the panoramic image.

其中，矫正图像是指对待处理的全景图像进行处理后得到的图像，用于代替待处理的全景图像作为后续实施例中全景图像处理方法的处理对象，以提高该全景图像处理方法的处理效果。The corrected image refers to an image obtained by processing the panoramic image to be processed, and is used to replace the panoramic image to be processed as the processing object of the panoramic image processing method in subsequent embodiments, so as to improve the processing effect of the panoramic image processing method.

如图2所示，在步骤S104中，通过可变形空间变换模型中的采样器(Sampler)来对全景图像进行像素矫正，得到全景图像对应的矫正图像。具体地，针对上述步骤S104所获得的映射像素位置与矫正像素位置之间的映射关系，通过采样器将全景图像中的映射像素位置的像素信息添加到相应的矫正像素位置上，从而得到全景图像对应的矫正图像。例如，全景图像中的人物和动物存在图像畸变，可先确定全景图像中与矫正图像中矫正的人和动物的像素位置分别对应的映射像素位置，再将映射像素位置的像素信息(比如颜色、透明度等)添加到矫正像素位置的像素信息中，最后得到矫正的人物和动物的全景图像。As shown in Figure 2, in step S104, the panoramic image is pixel corrected through a sampler (Sampler) in the deformable space transformation model to obtain a corrected image corresponding to the panoramic image. Specifically, for the mapping relationship between the mapped pixel position and the corrected pixel position obtained in the above step S104, the pixel information of the mapped pixel position in the panoramic image is added to the corresponding corrected pixel position through a sampler, thereby obtaining the panoramic image. Corresponding rectified image. For example, if people and animals in the panoramic image have image distortion, you can first determine the mapped pixel positions in the panoramic image corresponding to the pixel positions of the corrected people and animals in the corrected image, and then use the pixel information (such as color, color, etc.) of the mapped pixel positions to Transparency, etc.) is added to the pixel information of the corrected pixel position, and finally a corrected panoramic image of people and animals is obtained.

需要说明的是，如图2所示，本申请将空间变换(Spatial transformer)网络中的定位网络(localisation net)里的传统卷积替换成可变形卷积，构建得到可变形空间变换(Deformable spatial transformer)模型，可变形空间变换模型既可以增大卷积核的感受野，又可提高对全景图像中畸变对象的适应性，从而本申请能够更好地适应全景图像中不同位置的不同程度的畸变。其中，全景图像中的畸变对象指的是全景图像中存在成像扭曲或者成像变形的对象。例如，在全景图像中一面墙的成像可能出现墙面变宽、垂直的墙面发生弯曲等畸变情况，则该面墙可看作全景图像中的畸变对象。It should be noted that, as shown in Figure 2, this application replaces the traditional convolutions in the localization network (localization net) in the spatial transformer network with deformable convolutions to construct a deformable spatial transform (Deformable spatial transformer). transformer) model, the deformable spatial transformation model can not only increase the receptive field of the convolution kernel, but also improve the adaptability to distorted objects in panoramic images, so that this application can better adapt to different degrees of distortion at different positions in panoramic images. distortion. Among them, the distortion object in the panoramic image refers to the object with imaging distortion or imaging deformation in the panoramic image. For example, in a panoramic image, the image of a wall may be distorted by widening of the wall, bending of the vertical wall, etc., then the wall can be regarded as a distortion object in the panoramic image.

上述全景图像处理方法中，对待处理的全景图像进行可变形卷积处理，得到全景图像的特征图像，能够增强卷积过程的感受野，从全景图像中提取到更多的特征信息；进而对特征图像进行回归处理，得到全景图像的图像变换信息；根据图像变换信息和全景图像的矫正像素位置，得到全景图像中与矫正像素位置对应的映射像素位置；矫正像素位置表示为全景图像对应的矫正图像中像素的坐标位置，实现了全景图像的像素位置与矫正像素位置之间的映射关系确定；根据映射像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像，解决了全景图像的成像畸变的问题，通过上述全景图像处理方法对待处理的全景图像进行处理，有利于提升后续其他模型对全景图像的处理效果，而且全景图像处理过程能够灵活的与其他模型或网络结合，适用范围较广。In the above panoramic image processing method, the panoramic image to be processed is subjected to deformable convolution processing to obtain the characteristic image of the panoramic image, which can enhance the receptive field of the convolution process and extract more feature information from the panoramic image; and then extract the feature information The image is subjected to regression processing to obtain the image transformation information of the panoramic image; according to the image transformation information and the corrected pixel position of the panoramic image, the mapped pixel position corresponding to the corrected pixel position in the panoramic image is obtained; the corrected pixel position is expressed as the corrected image corresponding to the panoramic image The coordinate position of the pixel in the panoramic image realizes the determination of the mapping relationship between the pixel position of the panoramic image and the corrected pixel position; based on the pixel information of the mapped pixel position, the panoramic image is pixel corrected to obtain the corrected image corresponding to the panoramic image, which solves the problem of panoramic image To solve the problem of image distortion, using the above panoramic image processing method to process the panoramic image to be processed will help improve the subsequent processing effect of other models on panoramic images, and the panoramic image processing process can be flexibly combined with other models or networks, and is suitable for The scope is wider.

在一个实施例中，如图3所示，上述步骤S101，对待处理的全景图像进行可变形卷积处理，得到全景图像的特征图像，具体包括如下步骤：In one embodiment, as shown in Figure 3, the above-mentioned step S101 performs deformable convolution processing on the panoramic image to be processed to obtain the characteristic image of the panoramic image, which specifically includes the following steps:

步骤S301，根据全景图像的采样位置偏移量，对全景图像的采样位置进行偏移处理，得到全景图像的偏移后采样位置。Step S301: Perform offset processing on the sampling position of the panoramic image according to the offset amount of the sampling position of the panoramic image to obtain the offset sampling position of the panoramic image.

其中，采样位置偏移量指的是针对全景图像中每个像素的偏移方向信息；偏移方向信息表示像素朝设定方向偏移的距离，具体包括x轴方向和y轴方向；采样位置偏移量用于在不改变卷积核的情况下，改变卷积核的感受野范围。Among them, the sampling position offset refers to the offset direction information for each pixel in the panoramic image; the offset direction information represents the distance the pixel offsets in the set direction, specifically including the x-axis direction and the y-axis direction; the sampling position The offset is used to change the receptive field range of the convolution kernel without changing the convolution kernel.

其中，偏移后采样位置表示针对全景图像进行采样的采样点的位置信息。Among them, the offset sampling position represents the position information of the sampling point for sampling the panoramic image.

在图2的可变形定位网络(Deformable localisation net)中加入采样位置偏移量的学习之后，可变形卷积核的感受野的大小和位置可以根据全景图像中需要识别的对象进行动态调整。例如，传统的卷积核的感受野一般是3*3的形式，而采样位置偏移量能够使卷积核的感受野从3*3的正方形变成与全景图像中需要识别的对象相似的形状和大小。具体地，服务器对待处理的全景图像进行标准的卷积处理，得到全景图像的采样位置偏移量；由于采样位置偏移量包含全景图像中每个像素的偏移方向信息，所以采样位置偏移量的尺寸与待处理的全景图像的尺寸相同，可以将采样位置偏移量叠加到全景图像的各个像素的像素位置上，得到全景图像的偏移后采样位置。在实际应用中，将采样位置偏移量叠加在全景图像中的畸变对象上，得到畸变对象的偏移后采样位置，以便于卷积核对畸变对象的偏移后采样位置进行采样处理，使得卷积核能够采集到更多的畸变对象的像素信息，从而提高对全景图像中畸变对象的处理效果。After adding the learning of sampling position offset to the deformable localization network in Figure 2, the size and position of the receptive field of the deformable convolution kernel can be dynamically adjusted according to the objects that need to be recognized in the panoramic image. For example, the receptive field of a traditional convolution kernel is generally in the form of 3*3, and the sampling position offset can change the receptive field of the convolution kernel from a 3*3 square to one similar to the object that needs to be recognized in the panoramic image. Shape and size. Specifically, the server performs standard convolution processing on the panoramic image to be processed to obtain the sampling position offset of the panoramic image; since the sampling position offset contains the offset direction information of each pixel in the panoramic image, the sampling position offset The size of the amount is the same as the size of the panorama image to be processed, and the sampling position offset can be superimposed to the panorama image At the pixel position of each pixel, the offset sampling position of the panoramic image is obtained. In practical applications, the sampling position offset is superimposed on the distortion object in the panoramic image to obtain the offset sampling position of the distortion object, so that the convolution kernel can sample the offset sampling position of the distortion object, so that the convolution kernel The accumulation kernel can collect more pixel information of distorted objects, thereby improving the processing effect of distorted objects in panoramic images.

步骤S302，对偏移后采样位置进行线性插值处理，得到全景图像对应的偏移图像。Step S302: Perform linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image.

步骤S303，对偏移图像进行卷积处理，得到全景图像的特征图像。Step S303, perform convolution processing on the offset image to obtain the characteristic image of the panoramic image.

在一实施方式中，上述步骤S301所得到的全景图像的偏移后采样位置的数值可以是非整数，而且偏移后采样位置是指像素的坐标值，偏移后采样位置中不包含该坐标位置在图像中的像素信息；进而服务器对偏移后采样位置进行双线性插值处理，得到偏移后采样位置对应的目标采样位置；根据目标采样位置对应的像素信息，生成全景图像对应的偏移图像；使用卷积核对偏移图像进行卷积处理，得到全景图像的特征图像。In one embodiment, the value of the offset sampling position of the panoramic image obtained in the above step S301 may be a non-integer, and the offset sampling position refers to the coordinate value of the pixel, and the offset sampling position does not include the coordinate position. pixel information in the image; then the server performs bilinear interpolation processing on the offset sampling position to obtain the target sampling position corresponding to the offset sampling position; based on the pixel information corresponding to the target sampling position, the offset corresponding to the panoramic image is generated Image; use the convolution kernel to perform convolution processing on the offset image to obtain the characteristic image of the panoramic image.

本实施例中，通过采样位置偏移量对全景图像进行偏移处理，能够增强普通卷积核的感受野，从而提取到更多关于全景图像中的畸变对象的特征，有效的提高了全景图像的特征提取效果，有利于增强后续步骤中对全景图像中畸变对象的处理效果。In this embodiment, the offset processing of the panoramic image through the sampling position offset can enhance the receptive field of the ordinary convolution kernel, thereby extracting more features about the distorted objects in the panoramic image, effectively improving the performance of the panoramic image. The feature extraction effect is beneficial to enhancing the processing effect of distorted objects in panoramic images in subsequent steps.

在一个实施例中，在根据全景图像的采样位置偏移量，对全景图像的采样位置进行偏移处理，得到全景图像的偏移后采样位置之前，还包括：对全景图像进行卷积处理，得到全景图像的采样位置偏移量；采样位置偏移量的数量与卷积处理中的卷积核的数量相等。In one embodiment, before performing offset processing on the sampling position of the panoramic image according to the sampling position offset of the panoramic image to obtain the offset sampling position of the panoramic image, the method further includes: performing convolution processing on the panoramic image, Obtain the sampling position offset of the panoramic image; the number of sampling position offsets is equal to the number of convolution kernels in the convolution process.

需要说明的是，图2中可变形定位网络的第二次卷积处理可以是多通道输出，第二次卷积处理中卷积核的数量与通道数量相等，需要为第二次卷积处理中每一个卷积核都计算得到一个采样位置偏移量，从而利用采样位置偏移量去改变输入到第二次卷积处理的卷积核中的全景图像的像素的位置，以实现卷积核的感受野增大的目的。It should be noted that the second convolution process of the deformable positioning network in Figure 2 can be multi-channel output. The number of convolution kernels in the second convolution process is equal to the number of channels. It needs to be the second convolution process. Each convolution kernel in the calculation calculates a sampling position offset, thereby using the sampling position offset to change the position of the pixels of the panoramic image input to the convolution kernel of the second convolution process to achieve convolution The purpose of increasing the receptive field of the nucleus.

在一实施方式中，服务器对全景图像进行标准的卷积处理，得到全景图像中的畸变对象对应的采样位置偏移量，采样位置偏移量也可以通过梯度反向传播进行端到端的学习，在图2中的可变形空间变换模型的训练过程中，采样位置偏移量和传统的卷积处理中的卷积核同时更新。其中，由于采样位置偏移量中包含有全景图像的每个像素的偏移量，所以采样位置偏移量的尺寸与输入的全景图像的尺寸相同，得到的采样位置偏移量的数量与卷积处理中的卷积核的数量相等。In one embodiment, the server performs standard convolution processing on the panoramic image to obtain the sampling position offset corresponding to the distortion object in the panoramic image. The sampling position offset can also be learned end-to-end through gradient backpropagation. During the training process of the deformable spatial transformation model in Figure 2, the sampling position offset and the convolution kernel in traditional convolution processing are updated simultaneously. Among them, since the sampling position offset contains the offset of each pixel of the panoramic image, the size of the sampling position offset is the same as the size of the input panoramic image, and the number of obtained sampling position offsets is the same as the volume. The number of convolution kernels in product processing is equal.

本实施例中，通过对全景图像进行卷积处理，来获取采样位置偏移量，有利于后续步骤通过采样位置偏移量来增强卷积核的感受野，有助于提取到更多关于全景图像中的畸变对象的特征。In this embodiment, the sampling position offset is obtained by performing convolution processing on the panoramic image, which is beneficial to subsequent steps to enhance the receptive field of the convolution kernel through the sampling position offset, and helps to extract more information about the panorama. Characteristics of distorted objects in images.

在一个实施例中，上述步骤S102，对特征图像进行回归处理，得到全景图像的图像变换信息，具体包括如下内容：通过回归网络对特征图像进行仿射变换处理，得到全景图像的图像变换信息；回归网络是依据样本全景图像对待训练的回归网络进行训练得到。In one embodiment, the above step S102 performs regression processing on the feature image to obtain the image transformation information of the panoramic image, which specifically includes the following: performing affine transformation processing on the feature image through the regression network to obtain the image transformation information of the panoramic image; The regression network is trained based on the sample panoramic image of the regression network to be trained.

在一实施方式中，服务器在对待训练的回归网络训练的初始阶段，通过待训练的回归网络对样本全景图像进行仿射变换处理，会得到样本全景图像的恒等变换矩阵；然后根据恒等变换矩阵对待训练的回归网络进行梯度更新，得到回归网络。将特征图像再次输入到梯度更新后的回归网络中进行仿射变换处理，可以得到全景图像的仿射变换矩阵，即全景图像的图像变换信息。其中，仿射变换矩阵中包含有全景图像进行旋转、缩放和平移等处理的线性变换信息。In one embodiment, in the initial stage of training the regression network to be trained, the server performs affine transformation processing on the sample panoramic image through the regression network to be trained, and the identity transformation matrix of the sample panorama image is obtained; and then according to the identity transformation The matrix performs gradient updates on the regression network to be trained to obtain the regression network. Input the feature image into the gradient-updated regression network again for affine transformation processing to obtain the affine transformation matrix of the panoramic image, that is, the image transformation information of the panoramic image. Among them, the affine transformation matrix contains linear transformation information for rotation, scaling, and translation of the panoramic image.

需要说明的是，可变形卷积虽然能提升卷积核的感受野范围，但是对于全景图像来说，其感受野范围仍旧不够大，例如全景图像在靠近两侧的极端时畸变程度会特别大，相关像素的距离可能为100像素甚至100像素以上，采样位置偏移量无法让感受野覆盖相关像素，因而需要通过回归网络对特征图像进行全局处理，得到全景图像的图像变换信息。It should be noted that although deformable convolution can increase the receptive field range of the convolution kernel, its receptive field range is still not large enough for panoramic images. For example, the degree of distortion of panoramic images will be particularly large when approaching the extremes on both sides. , the distance between relevant pixels may be 100 pixels or even more than 100 pixels, and the sampling position offset cannot allow the receptive field to cover the relevant pixels. Therefore, the feature image needs to be globally processed through the regression network to obtain the image transformation information of the panoramic image.

本实施例中，通过回归网络对特征图像进行仿射变换处理，能够弥补可变形卷积的感受野无法顾及全景图像中的全局畸变的缺陷，使得本实施例中所获取的图像变换信息能够全面反映出全景图像中不同位置、不同程度的畸变，有助于提高全景图像的处理效果。In this embodiment, the regression network is used to perform affine transformation processing on the feature image, which can make up for the defect that the receptive field of the deformable convolution cannot take into account the global distortion in the panoramic image, so that the image transformation information obtained in this embodiment can comprehensively It reflects different positions and different degrees of distortion in the panoramic image, which helps to improve the processing effect of the panoramic image.

在一个实施例中，上述步骤S103，根据图像变换信息和全景图像的矫正像素位置，得到全景图像与矫正像素位置对应的映射像素位置，具体包括如下内容：将图像变换信息和矫正像素位置进行相乘，得到全景图像中与矫正像素位置对应的映射像素位置。In one embodiment, the above-mentioned step S103 obtains the mapped pixel position corresponding to the panoramic image and the corrected pixel position based on the image transformation information and the corrected pixel position of the panoramic image, which specifically includes the following content: comparing the image transformation information with the corrected pixel position. take, Obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image.

在一实施方式中，服务器对全景图像的图像变换信息和矫正像素位置进行矩阵运算，得到全景图像中与矫正像素位置对应的映射像素位置；建立矫正像素位置与映射像素位置之间的映射关系。获取全景图像中与矫正像素位置对应的映射像素位置可以通过如下公式进行表示：
In one embodiment, the server performs matrix operations on the image transformation information and corrected pixel positions of the panoramic image to obtain mapped pixel positions corresponding to the corrected pixel positions in the panoramic image; and establishes a mapping relationship between the corrected pixel positions and the mapped pixel positions. Obtaining the mapped pixel position corresponding to the corrected pixel position in the panoramic image can be expressed by the following formula:

其中，x_v和y_v分别表示矫正像素位置的x轴坐标和y轴坐标；θ₁₁、θ₁₂、θ₁₃、θ₂₁、θ₂₂和θ₂₃表示图像变换矩阵中的全景图像对应的各项图像变换信息；x_u和y_u分别表示与矫正像素位置对应映射像素位置x轴坐标和y轴坐标。Among them, x _v and y _v respectively represent the x-axis coordinate and y-axis coordinate of the corrected pixel position; θ ₁₁ , θ ₁₂ , θ ₁₃ , θ ₂₁ , θ ₂₂ and θ ₂₃ represent the items corresponding to the panoramic image in the image transformation matrix Image transformation information; x _u and y _u respectively represent the x-axis coordinate and y-axis coordinate of the mapped pixel position corresponding to the corrected pixel position.

本实施例中，通过可变形定位网络(Deformable localisation net)所输出的全景图像的图像变换信息和全景图像的矫正像素位置，可以实现将矫正像素位置和全景图像中的映射像素位置一一对应起来，以便于后续步骤中以映射像素位置为依据，对全景图像进行矫正。In this embodiment, through the image transformation information of the panoramic image output by the deformable localization network and the corrected pixel position of the panoramic image, a one-to-one correspondence between the corrected pixel position and the mapped pixel position in the panoramic image can be achieved , so that in subsequent steps, the panoramic image can be corrected based on the mapped pixel position.

在一个实施例中，上述步骤S104，根据映射像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像，具体包括如下内容：在映射像素位置的坐标值不是整数的情况下，对映射像素位置的坐标值进行线性插值处理，得到目标像素位置；目标像素位置的坐标值为整数；根据目标像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像。In one embodiment, the above-mentioned step S104 performs pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image, which specifically includes the following: When the coordinate value of the mapped pixel position is not an integer , perform linear interpolation processing on the coordinate value of the mapped pixel position to obtain the target pixel position; the coordinate value of the target pixel position is an integer; perform pixel correction on the panoramic image according to the pixel information of the target pixel position, and obtain the corrected image corresponding to the panoramic image.

在一实施方式中，服务器根据图像变换信息所确定的与矫正像素位置对应的映射像素位置的坐标值可能不是整数，在与矫正像素位置对应的映射像素位置的坐标值不是整数的情况下，是无法直接在全景图像中获取到相应的像素信息，进而服务器也无法确定需要放置在矫正像素位置的像素信息。因此，服务器需要对与矫正像素位置对应的映射像素位置的坐标值进行双线性插值处理，得到坐标值为整数的目标像素位置。In one embodiment, the coordinate value of the mapped pixel position corresponding to the corrected pixel position determined by the server based on the image transformation information may not be an integer. In the case where the coordinate value of the mapped pixel position corresponding to the corrected pixel position is not an integer, yes The corresponding pixel information cannot be obtained directly in the panoramic image, and the server cannot determine the pixel information that needs to be placed at the corrected pixel position. Therefore, the server needs to perform bilinear interpolation processing on the coordinate value of the mapped pixel position corresponding to the corrected pixel position to obtain the target pixel position whose coordinate value is an integer.

在一实施方式中，服务器将矫正像素位置对应的映射像素位置更新为目标像素位置，得到矫正像素位置和全景图像的目标像素位置之间的映射关系。根据目标像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像。In one embodiment, the server updates the mapped pixel position corresponding to the corrected pixel position to the target pixel position to obtain the mapping relationship between the corrected pixel position and the target pixel position of the panoramic image. According to the pixel information of the target pixel position, pixel correction is performed on the panoramic image to obtain a corrected image corresponding to the panoramic image.

举例说明，矫正像素位置为(3，4)，矫正像素位置对应的全景图像的像素位置为(1.2，5.6)，由于全景图像的像素位置通常是整数，所以在全景图像中无法查找到像素位置为(1.2，5.6)的像素信息，无法执行后续步骤得到矫正图像。假设通过双线性插值处理后得到的目标像素位置为(1，6)，进而根据全景图像中目标像素位置(1，6)的像素信息，对矫正像素位置(3，4)的像素信息进行更新，得到全景图像对应的矫正图像。For example, the corrected pixel position is (3, 4), and the pixel position of the panoramic image corresponding to the corrected pixel position is (1.2, 5.6). Since the pixel position of the panoramic image is usually an integer, the pixel position cannot be found in the panoramic image. The pixel information is (1.2, 5.6), and subsequent steps cannot be performed to obtain the corrected image. Assume that the target pixel position obtained by bilinear interpolation is (1, 6), and then based on the pixel information of the target pixel position (1, 6) in the panoramic image, the pixel information of the corrected pixel position (3, 4) is Update to obtain the corrected image corresponding to the panoramic image.

本实施例中，在映射像素位置的坐标值不是整数的情况下，对映射像素位置的坐标值进行线性插值处理，得到目标像素位置；然后根据目标像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像，实现了全景图像的目标像素位置与矫正图像中的矫正像素位置之间的映射关系的确定，有利于将全景图像的目标像素位置的像素信息更新到矫正图像的矫正像素位置中。In this embodiment, when the coordinate value of the mapped pixel position is not an integer, a linear interpolation process is performed on the coordinate value of the mapped pixel position to obtain the target pixel position; and then the panoramic image is corrected according to the pixel information of the target pixel position. , obtain the corrected image corresponding to the panoramic image, and realize the determination of the mapping relationship between the target pixel position of the panoramic image and the corrected pixel position in the corrected image, which is beneficial to updating the pixel information of the target pixel position of the panoramic image to the corrected image Corrected pixel position.

在一个实施例中，上述步骤S104，根据映射像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像，具体包括如下内容：在所述映射像素位置的坐标值是整数的情况下，将矫正像素位置的像素信息，更新为矫正像素位置对应的映射像素位置的像素信息，得到全景图像对应的矫正图像。In one embodiment, the above-mentioned step S104 performs pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image, which specifically includes the following content: the coordinate value of the mapped pixel position is an integer. In this case, the pixel information of the corrected pixel position is updated to the pixel information of the mapped pixel position corresponding to the corrected pixel position, and a corrected image corresponding to the panoramic image is obtained.

在一实施方式中，服务器通过采样器采集全景图像的映射像素位置或者目标像素位置的像素信息，然后将该像素信息填充到对应的矫正像素位置中；在所有的矫正像素位置均获取到像素信息之后，得到待处理的全景图像对应的矫正图像。In one embodiment, the server collects the pixel information of the mapped pixel position or the target pixel position of the panoramic image through a sampler, and then fills the pixel information into the corresponding corrected pixel position; pixel information is obtained at all corrected pixel positions. After that, the corrected image corresponding to the panoramic image to be processed is obtained.

本实施例中，通过将矫正像素位置的像素信息，更新为矫正像素位置对应的映射像素位置的像素信息，得到待处理的全景图像对应的矫正图像，解决了待处理的全景图像的成像畸变的问题，实现了对待处理的全景图像中的畸变成像进行矫正处理。In this embodiment, the pixel information of the corrected pixel position is updated to the pixel of the mapped pixel position corresponding to the corrected pixel position. Information is obtained to obtain the corrected image corresponding to the panoramic image to be processed, which solves the problem of imaging distortion of the panoramic image to be processed, and realizes the correction processing of the distorted image in the panoramic image to be processed.

在一个实施例中，在根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像之后，还包括如下内容：对矫正图像进行图像处理，得到矫正图像对应的目标全景图像。In one embodiment, after performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image, the following content is also included: performing image processing on the corrected image to obtain The target panoramic image corresponding to the rectified image.

在本实施例中，先通过对待处理的全景图像进行处理，能够得到待处理的全景图像对应的矫正图像，再让图像处理模型对矫正图像进行图像处理，在不改变图像处理模型的原有结构的同时，还能提升图像处理模型对矫正图像的处理效果，大大提升了得到的目标全景图像的质量。In this embodiment, the panoramic image to be processed is first processed to obtain the corrected image corresponding to the panoramic image to be processed, and then the image processing model is used to perform image processing on the corrected image without changing the original structure of the image processing model. At the same time, it can also improve the processing effect of the image processing model on the corrected image, greatly improving the quality of the obtained target panoramic image.

在一个实施例中，如图4所示，提供了一种全景图像处理方法，本实施例以该方法应用于服务器进行举例说明，可以理解的是，该方法也可以应用于终端，还可以应用于包括终端和服务器的***，并通过终端和服务器的交互实现。本实施例中，该方法包括以下步骤：In one embodiment, as shown in Figure 4, a panoramic image processing method is provided. This embodiment illustrates the application of this method to a server. It can be understood that this method can also be applied to terminals, and can also be applied to It is based on a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

步骤S401，获取待处理的全景图像对应的矫正图像。Step S401: Obtain the corrected image corresponding to the panoramic image to be processed.

其中，矫正图像通过上述步骤S101至S104实现，在此不再赘述。The corrected image is implemented through the above-mentioned steps S101 to S104, which will not be described again here.

步骤S402，对矫正图像进行图像处理，得到矫正图像对应的目标全景图像。Step S402: Perform image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image.

其中，图像处理可以是超分辨率处理，也可以是去噪处理，还可以是插帧处理，具体的图像处理的方式可以根据需求灵活变更，在此不进行具体限定。Among them, image processing can be super-resolution processing, denoising processing, or frame interpolation processing. The specific image processing method can be flexibly changed according to needs, and is not specifically limited here.

在一实施方式中，图5为上述全景图像处理方法的应用环境示意图，如图5所示，服务器获取待处理的全景图像；在对待处理的全景图像进行图像处理之前，先通过上述全景图像处理方法对待处理的全景图像进行处理，得到待处理的全景图像对应的矫正图像。进而服务器将矫正图像输入到其它的图像处理模型或者网络中进行图像处理，得到矫正图像对应的目标全景图像。In one embodiment, Figure 5 is a schematic diagram of the application environment of the above panoramic image processing method. As shown in Figure 5, the server obtains the panoramic image to be processed; before performing image processing on the panoramic image to be processed, the above panoramic image processing is performed The method processes the panoramic image to be processed and obtains the corrected image corresponding to the panoramic image to be processed. Then the server inputs the corrected image into other image processing models or networks for image processing, and obtains the target panoramic image corresponding to the corrected image.

本实施例中，由于全景图像具有畸变成像，使得图像处理模型在处理全景图像时图像处理效果较低，通过上述全景图像处理方法对待处理的全景图像进行处理，能够得到待处理的全景图像对应的矫正图像，让图像处理模型对矫正图像进行图像处理，在不改变图像处理模型的原有结构的同时，还能提升图像处理模型的处理效果，大大提升了得到的目标全景图像的质量。In this embodiment, since the panoramic image has a distorted image, the image processing effect of the image processing model is low when processing the panoramic image. By processing the panoramic image to be processed by the above panoramic image processing method, the panoramic image corresponding to the to be processed can be obtained. To rectify the image, let the image processing model perform image processing on the rectified image. Without changing the original structure of the image processing model, it can also improve the processing effect of the image processing model, greatly improving the quality of the obtained target panoramic image.

在一个实施例中，对矫正图像进行图像处理，得到矫正图像对应的目标全景图像，具体包括如下内容：对矫正图像进行超分处理，得到矫正图像对应的目标超分辨率全景图像；目标超分辨率全景图像的图像分辨率高于基于待处理的全景图像得到的超分辨率图像；或者，对矫正图像进行去噪处理，得到矫正图像对应的目标去噪全景图像；目标去噪全景图像包含的图像噪声少于基于待处理的全景图像得到的去噪后全景图像。In one embodiment, performing image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image specifically includes the following: performing super-resolution processing on the corrected image to obtain a target super-resolution panoramic image corresponding to the corrected image; target super-resolution The image resolution of the rate panoramic image is higher than the super-resolution image obtained based on the panoramic image to be processed; or, the rectified image is denoised to obtain the target denoised panoramic image corresponding to the rectified image; the target denoised panoramic image contains The image noise is less than the denoised panoramic image based on the panoramic image to be processed.

需要说明的是，通过上述步骤S101至S104所得到的矫正图像，可以输入到多种类型的图像处理模型中进行图像处理。例如，超分模型、去噪模型和插帧模型。It should be noted that the corrected image obtained through the above steps S101 to S104 can be input into various types of image processing models for image processing. For example, super-resolution model, denoising model and frame interpolation model.

在一实施方式中，可以将矫正图像输入到超分模型中进行超分处理，得到矫正图像对应的目标超分辨率全景图像，以提升矫正图像的分辨率；由于矫正图像预先经过了矫正处理，使得超分模型能够更准确的对矫正图像中的各个对象进行超分处理，而待处理的全景图像中畸变对象会影响超分模型的处理效果，所以目标超分辨率全景图像的图像分辨率高于基于待处理的全景图像得到的超分辨率图像。或者可以将矫正图像输入到去噪模型中进行去噪处理，得到矫正图像对应的目标去噪全景图像，以降低全景图像中的噪声；由于矫正图像预先经过了矫正处理，使得超分模型能够更准确的对矫正图像进行去噪处理，而待处理的全景图像中的畸变对象容易被去噪模型误认为是噪点，使得去噪模型对待处理的全景图像的去噪效果差于对矫正图像的去噪效果，所以目标去噪全景图像包含的图像噪声少于基于待处理的全景图像得到的去噪后全景图像。In one embodiment, the corrected image can be input into the super-resolution model for super-resolution processing to obtain a target super-resolution panoramic image corresponding to the corrected image, so as to improve the resolution of the corrected image; since the corrected image has been corrected in advance, This enables the super-resolution model to perform super-resolution processing on each object in the corrected image more accurately. However, the distorted objects in the panoramic image to be processed will affect the processing effect of the super-resolution model, so the image resolution of the target super-resolution panoramic image is high. Based on the super-resolution image obtained based on the panoramic image to be processed. Or you can input the corrected image into the denoising model for denoising processing to obtain the target denoised panoramic image corresponding to the corrected image to reduce the noise in the panoramic image; since the corrected image has been corrected in advance, the super-resolution model can be more accurate Accurately denoise the corrected image, and the distorted objects in the panoramic image to be processed are easily mistaken as noise points by the denoising model, making the denoising effect of the denoising model on the panoramic image to be processed worse than that on the corrected image. Noise effect, so the target denoised panoramic image contains less image noise than the denoised panoramic image based on the panoramic image to be processed.

本实施例中，通过上述全景图像处理方法得到的矫正图像，不仅能够与多种类型的图像处理模型结合使用，还无需改变图像处理模型的原有结构，在算力受限的场景中能够有效提升图像处理模型处理得到的目标全景图像的质量，使得本实施例中的全景图像处理方法适用范围较广，处理全景图像的效果较好。In this embodiment, the corrected image obtained through the above panoramic image processing method can not only be used in conjunction with multiple types of image processing models, but also does not need to change the original structure of the image processing model, and can be effectively used in scenarios with limited computing power. Improving the quality of the target panoramic image processed by the image processing model makes the panoramic image processing method in this embodiment have a wider application range and achieve better processing effects on the panoramic image.

在一个实施例中，如图6所示，提供了另一种全景图像处理方法，以该方法应用于服务器为例进行说明，包括以下步骤：In one embodiment, as shown in Figure 6, another panoramic image processing method is provided, and the method is applied to the server as an example. Line instructions, including the following steps:

步骤S601，对待处理的全景图像进行卷积处理，得到全景图像的采样位置偏移量；采样位置偏移量的数量与卷积处理中的卷积核的数量相等。Step S601: Convolve the panoramic image to be processed to obtain the sampling position offset of the panoramic image; the number of sampling position offsets is equal to the number of convolution kernels in the convolution process.

步骤S602，根据全景图像的采样位置偏移量，对全景图像的采样位置进行偏移处理，得到全景图像的偏移后采样位置。Step S602: Perform offset processing on the sampling position of the panoramic image according to the offset amount of the sampling position of the panoramic image to obtain the offset sampling position of the panoramic image.

步骤S603，对偏移后采样位置进行线性插值处理，得到全景图像对应的偏移图像；对偏移图像进行卷积处理，得到全景图像的特征图像。Step S603: Perform linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image; perform convolution processing on the offset image to obtain a characteristic image of the panoramic image.

步骤S604，通过回归网络对特征图像进行仿射变换处理，得到全景图像的图像变换信息；回归网络是依据样本全景图像对待训练的回归网络进行训练得到。Step S604: perform affine transformation processing on the feature image through a regression network to obtain the image transformation information of the panoramic image; the regression network is obtained by training the regression network to be trained based on the sample panoramic image.

步骤S605，将图像变换信息和矫正像素位置进行相乘，得到全景图像中与矫正像素位置对应的映射像素位置；矫正像素位置表示为全景图像对应的矫正图像中像素的坐标位置。可以理解的是，映射像素位置指的是全景图像中像素的坐标位置，矫正像素位置指的是矫正图像中像素的坐标位置，两种像素的坐标位置之间具有映射关系。Step S605: Multiply the image transformation information and the corrected pixel position to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is expressed as the coordinate position of the pixel in the corrected image corresponding to the panoramic image. It can be understood that the mapped pixel position refers to the coordinate position of the pixel in the panoramic image, and the corrected pixel position refers to the coordinate position of the pixel in the corrected image. There is a mapping relationship between the coordinate positions of the two pixels.

步骤S606-1，在映射像素位置的坐标值不是整数的情况下，将矫正像素位置的像素信息，更新为矫正像素位置对应的映射像素位置的像素信息，得到全景图像对应的矫正图像。Step S606-1: When the coordinate value of the mapped pixel position is not an integer, the pixel information of the corrected pixel position is updated to the pixel information of the mapped pixel position corresponding to the corrected pixel position, and a corrected image corresponding to the panoramic image is obtained.

步骤S606-2，在映射像素位置的坐标值不是整数的情况下，对映射像素位置的坐标值进行线性插值处理，得到目标像素位置；目标像素位置的坐标值为整数；根据目标像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像。Step S606-2, when the coordinate value of the mapped pixel position is not an integer, perform linear interpolation processing on the coordinate value of the mapped pixel position to obtain the target pixel position; the coordinate value of the target pixel position is an integer; according to the pixel at the target pixel position information, perform pixel correction on the panoramic image, and obtain the corrected image corresponding to the panoramic image.

可以理解的是，在步骤S606-1和步骤S606-2之前可以对全景图像中映射像素位置的坐标值是否为整数进行判断，从而根据判断结果确定执行步骤S606-1或步骤S606-2。It can be understood that before step S606-1 and step S606-2, it can be judged whether the coordinate value of the mapped pixel position in the panoramic image is an integer, so that step S606-1 or step S606-2 is executed according to the judgment result.

上述全景图像处理方法，具有以下有益效果：对待处理的全景图像进行可变形卷积处理，得到全景图像的特征图像，能够增强卷积过程的感受野，从全景图像中提取到更多的特征信息；进而对特征图像进行回归处理，得到全景图像的图像变换信息；根据图像变换信息和全景图像的矫正像素位置，得到全景图像中与矫正像素位置对应的映射像素位置；矫正像素位置表示为全景图像对应的矫正图像中像素的坐标位置，实现了全景图像的像素位置与矫正像素位置之间的映射关系确定；根据映射像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像，解决了全景图像的成像畸变的问题，通过上述全景图像处理方法对待处理的全景图像进行处理，有利于提升其他模型对全景图像的处理效果，而且全景图像处理过程能够灵活的与其他模型或网络结合，适用范围较广。The above panoramic image processing method has the following beneficial effects: the panoramic image to be processed is subjected to deformable convolution processing to obtain the characteristic image of the panoramic image, which can enhance the receptive field of the convolution process and extract more feature information from the panoramic image. ; Then perform regression processing on the feature image to obtain the image transformation information of the panoramic image; according to the image transformation information and the corrected pixel position of the panoramic image, obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is expressed as the panoramic image The corresponding coordinate position of the pixel in the corrected image realizes the determination of the mapping relationship between the pixel position of the panoramic image and the corrected pixel position; based on the pixel information of the mapped pixel position, pixel correction is performed on the panoramic image to obtain the corrected image corresponding to the panoramic image , solves the problem of imaging distortion of panoramic images, and processing the panoramic images to be processed through the above panoramic image processing method is conducive to improving the processing effect of other models on panoramic images, and the panoramic image processing process can be flexibly integrated with other models or networks Combined, the scope of application is wider.

为了更清晰阐明本公开实施例提供的全景图像处理方法，以下以一个具体的实施例对该全景图像处理方法进行具体说明。在一个实施例中，提供了又一种全景图像处理方法，可以应用于如图2所示的应用环境中，具体包括如下内容：In order to more clearly illustrate the panoramic image processing method provided by the embodiments of the present disclosure, the panoramic image processing method will be described in detail below using a specific embodiment. In one embodiment, another panoramic image processing method is provided, which can be applied in the application environment as shown in Figure 2, specifically including the following:

先通过一个卷积层对待处理的全景图像进行卷积处理，得到待处理的全景图像的采样位置偏移量，将学***移等一切线性变换信息。对全景图像的图像变换矩阵和全景图像的矫正像素位置进行矩阵运算，得到全景图像中与矫正像素位置对应的映射像素位置，最后将该映射像素位置的像素信息填充到对应的矫正像素位置的像素信息中，得到全景图像对应的矫正图像。First, perform convolution processing on the panoramic image to be processed through a convolution layer to obtain the sampling position offset of the panoramic image to be processed, and add the learned sampling position offset to the original input panoramic image to be processed. The offset image of the panoramic image to be processed is obtained through bilinear interpolation, and then the offset image is subjected to ordinary convolution to obtain the feature image; the feature image is matrix transformed through the regression network to obtain the image transformation matrix; after deformable convolution While the product increases the receptive field of the convolution kernel, it can also improve the adaptability to different positions and degrees of distortion in the panoramic image through matrix transformation. Among them, the image transformation matrix is an identity transformation matrix at the initial stage. After training and learning gradient update, the image transformation matrix will become an affine transformation matrix. The affine transformation matrix includes the rotation, scaling, translation, etc. of the panoramic image. All linear transformation information. Perform matrix operations on the image transformation matrix of the panoramic image and the corrected pixel position of the panoramic image to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image, and finally fill the pixel information of the mapped pixel position into the pixel of the corresponding corrected pixel position In the information, the corrected image corresponding to the panoramic image is obtained.

在本实施例中，针对全景图片畸变这一特性，提出全景图像处理方法，可广泛用于任何模型或者网络对全景图像的图像处理之前，从而提升模型或者网络的图像处理效果，而无需对原模型或者网络进行改动，只需在对全景图像的图像处理之前运用本全景图像处理方法即可，操作方便的同时也可以带来效果的提升，适用范围广。In this embodiment, a panoramic image processing method is proposed for the characteristic of panoramic image distortion, which can be widely used before image processing of panoramic images by any model or network, thereby improving the image processing effect of the model or network without the need to modify the original image. To make changes to the model or network, you only need to apply this panoramic image processing method before image processing of the panoramic image. It is easy to operate and can also improve the effect and has a wide range of applications.

应该理解的是，虽然如上所述的各实施例所涉及的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，如上所述的各实施例所涉及的流程图中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。It should be understood that although the various steps in the flowcharts involved in the various embodiments described above are shown in sequence as indicated by arrows, However, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments may include multiple steps or stages. These steps or stages are not necessarily executed at the same time, but may be completed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

在一个实施例中，提供了一种计算机设备，该计算机设备可以是服务器，其内部结构图可以如图7所示。该计算机设备包括通过***总线连接的处理器、存储器和网络接口。其中，该计算机设备的处理器配置为提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质和内存储器。该非易失性存储介质存储有操作***、计算机程序和数据库。该内存储器为非易失性存储介质中的操作***和计算机程序的运行提供环境。该计算机设备的数据库配置为存储待处理的全景图像、矫正图像和目标全景图像等数据。该计算机设备的网络接口配置为与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种全景图像处理方法或者一种全景图像处理方法。In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in Figure 7 . The computer device includes a processor, memory, and network interfaces connected through a system bus. Wherein, the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage medium stores operating systems, computer programs and databases. This internal memory provides an environment for the execution of operating systems and computer programs in non-volatile storage media. The database of the computer device is configured to store data such as panoramic images to be processed, rectified images, and target panoramic images. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program implements a panoramic image processing method or a panoramic image processing method when executed by the processor.

本领域技术人员可以理解，图7中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应配置为其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。Those skilled in the art can understand that the structure shown in Figure 7 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment that should be configured on the solution of the present application. The specific computer equipment More or fewer components may be included than shown in the figures, or certain components may be combined, or may have a different arrangement of components.

在一个实施例中，还提供了一种计算机设备，包括存储器和处理器，存储器中存储有计算机程序，该处理器执行计算机程序时实现上述各方法实施例中的步骤。In one embodiment, a computer device is also provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements the steps in the above method embodiments.

在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述各方法实施例中的步骤。In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

在一个实施例中，提供了一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现上述各方法实施例中的步骤。In one embodiment, a computer program product is provided, including a computer program that implements the steps in each of the above method embodiments when executed by a processor.

需要说明的是，本申请所涉及的用户信息(包括但不限于用户设备信息、用户个人信息等)和数据(包括但不限于用于分析的数据、存储的数据、展示的数据和图像数据等)，均为经用户授权或者经过各方充分授权的信息和数据。It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, image data, etc.) involved in this application ), are all information and data authorized by the user or fully authorized by all parties.

本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、数据库或其它介质的任何引用，均可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-Only Memory，ROM)、磁带、软盘、闪存、光存储器、高密度嵌入式非易失性存储器、阻变存储器(ReRAM)、磁变存储器(Magnetoresistive Random Access Memory，MRAM)、铁电存储器(Ferroelectric Random Access Memory，FRAM)、相变存储器(Phase Change Memory，PCM)、石墨烯存储器等。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或外部高速缓冲存储器等。作为说明而非局限，RAM可以是多种形式，比如静态随机存取存储器(Static Random Access Memory，SRAM)或动态随机存取存储器(Dynamic Random Access Memory，DRAM)等。本申请所提供的各实施例中所涉及的数据库可包括关系型数据库和非关系型数据库中至少一种。非关系型数据库可包括基于区块链的分布式数据库等，不限于此。本申请所提供的各实施例中所涉及的处理器可为通用处理器、中央处理器、图形处理器、数字信号处理器、可编程逻辑器、基于量子计算的数据处理逻辑器等，不限于此。Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage. In the media, when executed, the computer program may include the processes of the above method embodiments. Any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory (MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in many forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include blockchain-based distributed databases, etc., but are not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to this.

以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对本申请专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请的保护范围应以所附权利要求为准。 The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the scope of protection of this application should be determined by the appended claims.

工业实用性Industrial applicability

本申请实施例所提供的全景图像处理方法、计算机设备、存储介质和计算机程序产品，对待处理的全景图像进行可变形卷积处理，得到全景图像的特征图像，能够增强卷积过程的感受野，从全景图像中提取到更多的特征信息；进而对特征图像进行回归处理，得到全景图像的图像变换信息；根据图像变换信息和全景图像的矫正像素位置，得到全景图像中与矫正像素位置对应的映射像素位置；矫正像素位置表示为全景图像对应的矫正图像中像素的坐标位置，实现了全景图像的像素位置与矫正像素位置之间的映射关系确定；根据映射像素位置的像素信息，对全景图像进行像素矫正，得到全景图像对应的矫正图像，解决了全景图像的成像畸变的问题，通过上述全景图像处理方法对待处理的全景图像进行处理，有利于提升后续其他模型对全景图像的处理效果，而且全景图像处理过程能够灵活的与其他模型或网络结合，适用范围较广。 The panoramic image processing method, computer equipment, storage medium and computer program product provided by the embodiments of the present application can perform deformable convolution processing on the panoramic image to be processed to obtain the characteristic image of the panoramic image, which can enhance the receptive field of the convolution process. Extract more feature information from the panoramic image; then perform regression processing on the feature image to obtain the image transformation information of the panoramic image; according to the image transformation information and the corrected pixel position of the panoramic image, obtain the corrected pixel position in the panoramic image corresponding to Mapping the pixel position; the corrected pixel position is expressed as the coordinate position of the pixel in the corrected image corresponding to the panoramic image, realizing the determination of the mapping relationship between the pixel position of the panoramic image and the corrected pixel position; according to the pixel information of the mapped pixel position, the panoramic image is Perform pixel correction to obtain the corrected image corresponding to the panoramic image, which solves the problem of imaging distortion of the panoramic image. Processing the panoramic image to be processed through the above panoramic image processing method will help improve the subsequent processing effect of other models on the panoramic image, and The panoramic image processing process can be flexibly combined with other models or networks and has a wide range of applications.

Claims

一种全景图像处理方法，所述方法包括：A panoramic image processing method, the method includes:

对待处理的全景图像进行可变形卷积处理，得到所述全景图像的特征图像；Perform deformable convolution processing on the panoramic image to be processed to obtain a characteristic image of the panoramic image;

对所述特征图像进行回归处理，得到所述全景图像的图像变换信息；Perform regression processing on the feature image to obtain image transformation information of the panoramic image;

根据所述图像变换信息和所述全景图像的矫正像素位置，得到所述全景图像中与所述矫正像素位置对应的映射像素位置；所述矫正像素位置表示为所述全景图像对应的矫正图像中像素的坐标位置；According to the image transformation information and the corrected pixel position of the panoramic image, the mapped pixel position corresponding to the corrected pixel position in the panoramic image is obtained; the corrected pixel position is expressed as the corrected pixel position corresponding to the panoramic image. The coordinate position of the pixel;

根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像。According to the pixel information of the mapped pixel position, pixel correction is performed on the panoramic image to obtain a corrected image corresponding to the panoramic image.
根据权利要求1所述的方法，其中，所述对待处理的全景图像进行可变形卷积处理，得到所述全景图像的特征图像，包括：The method according to claim 1, wherein the panoramic image to be processed is subjected to deformable convolution processing to obtain the characteristic image of the panoramic image, including:

根据所述全景图像的采样位置偏移量，对所述全景图像的采样位置进行偏移处理，得到所述全景图像的偏移后采样位置；According to the sampling position offset of the panoramic image, perform offset processing on the sampling position of the panoramic image to obtain the offset sampling position of the panoramic image;

对所述偏移后采样位置进行线性插值处理，得到所述全景图像对应的偏移图像；Perform linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image;

对所述偏移图像进行卷积处理，得到所述全景图像的特征图像。Convolution processing is performed on the offset image to obtain a characteristic image of the panoramic image.
根据权利要求2所述的方法，其中，在根据所述全景图像的采样位置偏移量，对所述全景图像的采样位置进行偏移处理，得到所述全景图像的偏移后采样位置之前，还包括：The method according to claim 2, wherein before performing offset processing on the sampling position of the panoramic image according to the sampling position offset of the panoramic image to obtain the offset sampling position of the panoramic image, Also includes:

对所述全景图像进行卷积处理，得到所述全景图像的采样位置偏移量；所述采样位置偏移量的数量与所述卷积处理中的卷积核的数量相等。Perform convolution processing on the panoramic image to obtain a sampling position offset of the panoramic image; the number of sampling position offsets is equal to the number of convolution kernels in the convolution process.
根据权利要求1所述的方法，其中，所述对所述特征图像进行回归处理，得到所述全景图像的图像变换信息，包括：The method according to claim 1, wherein performing regression processing on the feature image to obtain image transformation information of the panoramic image includes:

通过回归网络对所述特征图像进行仿射变换处理，得到所述全景图像的图像变换信息；所述回归网络是依据样本全景图像对待训练的回归网络进行训练得到。The feature image is subjected to affine transformation processing through a regression network to obtain the image transformation information of the panoramic image; the regression network is obtained by training the regression network to be trained based on the sample panoramic image.
根据权利要求1所述的方法，其中，所述根据所述图像变换信息和所述全景图像的矫正像素位置，得到所述全景图像中与所述矫正像素位置对应的映射像素位置，包括：The method according to claim 1, wherein obtaining the mapped pixel position corresponding to the corrected pixel position in the panoramic image based on the image transformation information and the corrected pixel position of the panoramic image includes:

将所述图像变换信息和所述矫正像素位置进行相乘，得到所述全景图像中与所述矫正像素位置对应的映射像素位置。The image transformation information and the corrected pixel position are multiplied to obtain a mapped pixel position corresponding to the corrected pixel position in the panoramic image.
根据权利要求1所述的方法，其中，所述根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像，包括：The method according to claim 1, wherein performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image includes:

在所述映射像素位置的坐标值不是整数的情况下，对所述映射像素位置的坐标值进行线性插值处理，得到目标像素位置；所述目标像素位置的坐标值为整数；When the coordinate value of the mapped pixel position is not an integer, perform linear interpolation processing on the coordinate value of the mapped pixel position to obtain the target pixel position; the coordinate value of the target pixel position is an integer;

根据所述目标像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像。According to the pixel information of the target pixel position, pixel correction is performed on the panoramic image to obtain a corrected image corresponding to the panoramic image.
根据权利要求1所述的方法，其中，所述根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像，包括：The method according to claim 1, wherein performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image includes:

在所述映射像素位置的坐标值是整数的情况下，将所述矫正像素位置的像素信息，更新为所述矫正像素位置对应的映射像素位置的像素信息，得到所述全景图像对应的矫正图像。When the coordinate value of the mapped pixel position is an integer, the pixel information of the corrected pixel position is updated to the pixel information of the mapped pixel position corresponding to the corrected pixel position, and a corrected image corresponding to the panoramic image is obtained. .
根据权利要求1-7任一项所述的方法，其中，还包括：The method according to any one of claims 1-7, further comprising:

对所述矫正图像进行图像处理，得到所述矫正图像对应的目标全景图像。Perform image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image.
根据权利要求8所述的方法，其中，所述对所述矫正图像进行图像处理，得到所述矫正图像对应的目标全景图像，包括：The method according to claim 8, wherein said performing image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image includes:

对所述矫正图像进行超分处理，得到所述矫正图像对应的目标超分辨率全景图像；所述目标超分辨率全景图像的图像分辨率高于基于所述待处理的全景图像得到的超分辨率图像；Perform super-resolution processing on the corrected image to obtain a target super-resolution panoramic image corresponding to the corrected image; the image resolution of the target super-resolution panoramic image is higher than the super-resolution obtained based on the panoramic image to be processed. rate image;

或者，or,

对所述矫正图像进行去噪处理，得到所述矫正图像对应的目标去噪全景图像；所述目标去噪全景图像包含的图像噪声少于基于所述待处理的全景图像得到的去噪后全景图像。 Perform denoising processing on the corrected image to obtain a target denoised panoramic image corresponding to the corrected image; the target denoised panoramic image contains less image noise than the denoised panorama obtained based on the panoramic image to be processed. image.
一种全景图像处理装置，包括：A panoramic image processing device, including:

卷积处理单元，被配置为执行对待处理的全景图像进行可变形卷积处理，得到所述全景图像的特征图像；a convolution processing unit configured to perform deformable convolution processing on the panoramic image to be processed to obtain a characteristic image of the panoramic image;

回归处理单元，被配置为执行对所述特征图像进行回归处理，得到所述全景图像的图像变换信息；A regression processing unit configured to perform regression processing on the feature image to obtain image transformation information of the panoramic image;

像素映射单元，被配置为执行根据所述图像变换信息和所述全景图像的矫正像素位置，得到所述全景图像中与所述矫正像素位置对应的映射像素位置；所述矫正像素位置表示为所述全景图像对应的矫正图像中像素的坐标位置；a pixel mapping unit configured to perform a correction pixel position of the panoramic image according to the image transformation information and obtain a mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is represented by The coordinate positions of the pixels in the corrected image corresponding to the panoramic image;

像素矫正单元，被配置为执行根据所述映射像素位置的像素信息，对所述全景图像进行像素矫正，得到所述全景图像对应的矫正图像。The pixel correction unit is configured to perform pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image.
根据权利要求10所述的装置，其中，所述卷积处理单元，还被配置为执行：The device according to claim 10, wherein the convolution processing unit is further configured to perform:

根据所述全景图像的采样位置偏移量，对所述全景图像的采样位置进行偏移处理，得到所述全景图像的偏移后采样位置；According to the sampling position offset of the panoramic image, perform offset processing on the sampling position of the panoramic image to obtain the offset sampling position of the panoramic image;

对所述偏移后采样位置进行线性插值处理，得到所述全景图像对应的偏移图像；Perform linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image;

对所述偏移图像进行卷积处理，得到所述全景图像的特征图像。Convolution processing is performed on the offset image to obtain a characteristic image of the panoramic image.
根据权利要求10所述的装置，其中，所述回归处理单元，还被配置为执行：The device according to claim 10, wherein the regression processing unit is further configured to perform:

通过回归网络对所述特征图像进行仿射变换处理，得到所述全景图像的图像变换信息；所述回归网络是依据样本全景图像对待训练的回归网络进行训练得到。The feature image is subjected to affine transformation processing through a regression network to obtain the image transformation information of the panoramic image; the regression network is obtained by training the regression network to be trained based on the sample panoramic image.
一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现权利要求1至9中任一项所述的方法的步骤。A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1至9中任一项所述的方法的步骤。A computer-readable storage medium having a computer program stored thereon, which implements the steps of the method according to any one of claims 1 to 9 when executed by a processor.
一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现权利要求1至9中任一项所述的方法的步骤。 A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.