WO2022160857A1 - Image processing method and apparatus, and computer-readable storage medium and electronic device - Google Patents
Image processing method and apparatus, and computer-readable storage medium and electronic device Download PDFInfo
- Publication number
- WO2022160857A1 WO2022160857A1 PCT/CN2021/129615 CN2021129615W WO2022160857A1 WO 2022160857 A1 WO2022160857 A1 WO 2022160857A1 CN 2021129615 W CN2021129615 W CN 2021129615W WO 2022160857 A1 WO2022160857 A1 WO 2022160857A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- starry sky
- sky area
- original image
- stars
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 55
- 230000004927 fusion Effects 0.000 claims abstract description 51
- 238000003709 image segmentation Methods 0.000 claims abstract description 22
- 239000011159 matrix material Substances 0.000 claims description 25
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 230000011218 segmentation Effects 0.000 claims description 13
- 230000009466 transformation Effects 0.000 claims description 11
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 52
- 230000000694 effects Effects 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 38
- 238000010586 diagram Methods 0.000 description 17
- 238000000605 extraction Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000001514 detection method Methods 0.000 description 8
- 241000475481 Nebula Species 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 238000005282 brightening Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/02—Affine transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present disclosure relates to the technical field of image processing, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device.
- a star map downloaded from the network may be used as prior information to assist in shooting a starry sky. Specifically, the corresponding star map is found from the network through the shooting time and location of the image. Next, the area on the star map corresponding to the shot image is determined, and then the image is adjusted according to the grayscale of the stars in the star map area. Grayscale of stars.
- the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
- a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
- the star closest to the star in the original image may be determined from the starry sky region of the reference image. For example, if the position of star A in the original image is (x, y), then it corresponds to the reference image, and the position of (x, y) in the reference image is used as the benchmark to determine the closest star A' to this distance, as the The star corresponding to star A in the original image.
- different feature extraction algorithms may be used for the extraction process of corner features, and these feature extraction algorithms may include but are not limited to FAST feature point detection algorithm, DOG feature point detection algorithm, Harris feature point detection algorithm Detection algorithm, SIFT feature point detection algorithm, SURF feature point detection algorithm, etc.
- Feature descriptors may include, but are not limited to, BRIEF feature point descriptors, BRISK feature point descriptors, FREAK feature point descriptors, and the like.
- a homography matrix for a non-starry sky area may be estimated by random sampling consistency as the second homography matrix.
- the first intermediate image and the second intermediate image may be combined into a matching image corresponding to the original image. standard image.
- the image patch to be fused can be determined from the registered image and the original image.
- the statistical histogram is represented by the proportion of pixels
- the statistical histogram curve of the non-star pixels in the starry sky area in the second target image is curve 1
- the statistical histogram curve of the non-star pixels in the starry sky area in the third target image is curve 2.
- the curve of the first fusion weight in the embodiment of the present disclosure may be constructed as the curve 3
- the curve of the second fusion weight may be constructed as the curve 4 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
An image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device, which relate to the technical field of image processing. The image processing method comprises: performing image segmentation on an original image, so as to determine a starry sky area and a non-starry sky area of the original image (S32); by means of a starry sky area of a reference image and the starry sky area of the original image, determining a first intermediate image corresponding to the starry sky area of the original image (S34); by means of a non-starry sky area of the reference image and the non-starry sky area of the original image, determining a second intermediate image corresponding to the non-starry sky area of the original image (S36); and performing image fusion on the basis of the first intermediate image, the second intermediate image and the original image, so as to generate a first target image (S38). By means of the method, the display effect of an image including a starry sky can be improved.
Description
相关申请的交叉引用CROSS-REFERENCE TO RELATED APPLICATIONS
本申请要求于2021年01月28日提交的申请号为202110120725.4、名称为“图像处理方法及装置、计算机可读存储介质和电子设备”的中国专利申请的优先权，该中国专利申请的全部内容通过引用全部并入本文。This application claims the priority of the Chinese patent application with the application number 202110120725.4 and the title of "Image Processing Method and Device, Computer-readable Storage Medium and Electronic Equipment" filed on January 28, 2021, the entire content of the Chinese patent application Incorporated herein by reference in its entirety.
本公开涉及图像处理技术领域，具体而言，涉及一种图像处理方法、图像处理装置、计算机可读存储介质和电子设备。The present disclosure relates to the technical field of image processing, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device.
随着带有拍摄功能的电子设备的普及，越来越多的用户使用电子设备来获取场景图像。拍摄场景已不限于光线较好的场景，用户也希望能够拍摄出效果较好的包含星空的图像。With the popularization of electronic devices with photographing functions, more and more users use electronic devices to acquire scene images. The shooting scene is not limited to the scene with better light, and users also hope to be able to take a better image including the starry sky.
然而，由于星体之间的相对运动、电子设备相机抖动等原因，往往会出现包含星空的图像显示效果不佳的问题。However, due to relative motion between stars, camera shake of electronic equipment, etc., there is often a problem that images containing stars are not displayed well.
发明内容SUMMARY OF THE INVENTION
根据本公开的第一方面，提供了一种图像处理方法，包括：对原始图像进行图像分割，确定出原始图像的星空区域和非星空区域；通过参考图像的星空区域和原始图像的星空区域，确定出与原始图像的星空区域对应的第一中间图像；通过参考图像的非星空区域和原始图像的非星空区域，确定出原始图像的非星空区域对应的第二中间图像；基于第一中间图像、第二中间图像和原始图像，进行图像融合，并生成第一目标图像。According to a first aspect of the present disclosure, an image processing method is provided, including: performing image segmentation on an original image to determine a starry sky area and a non-starry sky area of the original image; by referring to the starry sky area of the image and the starry sky area of the original image, Determine the first intermediate image corresponding to the starry sky area of the original image; determine the second intermediate image corresponding to the non-starry sky area of the original image by referring to the non-starry sky area of the original image and the non-starry sky area of the original image; based on the first intermediate image , the second intermediate image and the original image, perform image fusion, and generate a first target image.
根据本公开的第二方面，提供了一种图像处理装置，包括：图像分割模块，被配置为对原始图像进行图像分割，确定出原始图像的星空区域和非星空区域；第一图像确定模块，被配置为通过参考图像的星空区域和原始图像的星空区域，确定出与原始图像的星空区域对应的第一中间图像；第二图像确定模块，被配置为通过参考图像的非星空区域和原始图像的非星空区域，确定出原始图像的非星空区域对应的第二中间图像；第一图像生成模块，被配置为基于第一中间图像、第二中间图像和原始图像，进行图像融合，并生成第一目标图像。According to a second aspect of the present disclosure, there is provided an image processing apparatus, comprising: an image segmentation module configured to perform image segmentation on an original image to determine a starry sky area and a non-starry sky area of the original image; a first image determination module, is configured to determine the first intermediate image corresponding to the starry sky area of the original image by using the starry sky area of the reference image and the starry sky area of the original image; the second image determining module is configured to pass the non-starry sky area of the reference image and the original image The non-starry sky area of the original image is determined, and the second intermediate image corresponding to the non-starry sky area of the original image is determined; the first image generation module is configured to perform image fusion based on the first intermediate image, the second intermediate image and the original image, and generate the first intermediate image. a target image.
根据本公开的第三方面，提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述的图像处理方法。According to a third aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the above-mentioned image processing method.
根据本公开的第四方面，提供了一种电子设备，包括处理器；存储器，被配置为存储一个或多个程序，当一个或多个程序被处理器执行时，使得所述处理器实现上述的图像处理方法。According to a fourth aspect of the present disclosure, there is provided an electronic device, comprising a processor; a memory configured to store one or more programs, which, when executed by the processor, cause the processor to implement the above-mentioned image processing method.
图1示出了本公开实施例的图像处理方案的示例性***架构的示意图；FIG. 1 shows a schematic diagram of an exemplary system architecture of an image processing solution according to an embodiment of the present disclosure;
图2示出了适于用来实现本公开实施例的电子设备的结构示意图；FIG. 2 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure;
图3示意性示出了根据本公开示例性实施方式的图像处理方法的流程图；FIG. 3 schematically shows a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;
图4示出了本公开实施例的生成星空区域和非星空区域的掩膜的过程示意图；4 shows a schematic diagram of a process for generating masks for starry sky areas and non-starry sky areas according to an embodiment of the present disclosure;
图5示意性示出了本公开实施例的对原始图像的星空区域进行配准处理的流程图；5 schematically shows a flow chart of performing registration processing on a starry sky area of an original image according to an embodiment of the present disclosure;
图6示意性示出了本公开实施例的对原始图像的非星空区域进行配准处理的流程图；FIG. 6 schematically shows a flow chart of performing registration processing on a non-starry sky area of an original image according to an embodiment of the present disclosure;
图7示出了本公开实施例的图像融合以生成第一目标图像的示意图；FIG. 7 shows a schematic diagram of image fusion to generate a first target image according to an embodiment of the present disclosure;
图8示出了本公开实施例的第一融合权值和第二融合权值的示意图；FIG. 8 shows a schematic diagram of a first fusion weight and a second fusion weight in an embodiment of the present disclosure;
图9示意性示出了本公开实施例的图像处理方法的整个过程的流程图；FIG. 9 schematically shows a flowchart of the entire process of the image processing method according to the embodiment of the present disclosure;
图10示意性示出了根据本公开示例性实施方式的图像处理装置的方框图；FIG. 10 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure;
图11示意性示出了根据本公开另一示例性实施方式的图像处理装置的方框图；FIG. 11 schematically shows a block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure;
图12示意性示出了根据本公开又一示例性实施方式的图像处理装置的方框图。FIG. 12 schematically shows a block diagram of an image processing apparatus according to yet another exemplary embodiment of the present disclosure.
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本公开的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本公开的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、组元、装置、步骤等。在其它情况下，不详细示出或描述公知技术方案以避免喧宾夺主而使得本公开的各方面变得模糊。Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the figures are functional entities that do not necessarily necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
附图中所示的流程图仅是示例性说明，不是必须包括所有的步骤。例如，有的步骤还可以分解，而有的步骤可以合并或部分合并，因此实际执行的顺序有可能根据实际情况改变。另外，下面所有的术语“第一”、“第二”、“第三”、“第四”仅是为了区分的目的，不应作为本公开内容的限制。The flow charts shown in the figures are merely illustrative and do not necessarily include all steps. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may be changed according to the actual situation. In addition, all the following terms "first", "second", "third", and "fourth" are for the purpose of distinction only and should not be used as a limitation of the present disclosure.
终端设备的相机拍摄出的包含星空的图像通常很暗，因此需要给相机的传感器(sensor)较长的曝光时间或设置较大的ISO才可能正常曝光。但是，因为星体之间的相对运动，星空相对终端设备在缓慢运动。在这种情况下，当曝光时间长于某个阈值(该阈值与相机镜头焦距相关，例如手机主摄约为20秒)后，星星会在sensor上形成明显的星轨，银河星云将变得模糊不清，影响星空照片的美观性。The image containing the starry sky shot by the camera of the terminal device is usually very dark, so it is necessary to give the camera's sensor (sensor) a longer exposure time or set a larger ISO to allow normal exposure. However, due to the relative motion between the stars, the stars are moving slowly relative to the terminal device. In this case, when the exposure time is longer than a certain threshold (the threshold is related to the focal length of the camera lens, for example, the main camera of the mobile phone is about 20 seconds), the stars will form obvious star trails on the sensor, and the Milky Way nebula will become blurred Unclear, affecting the beauty of starry sky photos.
因此，在拍摄星空时，曝光时间需要限定在会产生星轨的阈值以下，同时不得不采用很高的ISO以曝光出相对效果较好的图像。Therefore, when shooting a starry sky, the exposure time needs to be limited below the threshold that will produce star trails, and at the same time, a very high ISO has to be used to expose images with relatively good effects.
然而，终端设备(例如手机、平板电脑等)上的sensor面积较小，在采用上述方案拍摄星空时，图像中会出现很多噪声。同时，sensor的感光性可能不强，在较短的曝光时间内，可能无法把大部分星星和银河星云拍摄得很清楚，所以无法拍出显示效果较好的星空图像。However, the area of sensors on terminal devices (such as mobile phones, tablet computers, etc.) is small, and when the above scheme is used to photograph the starry sky, there will be a lot of noise in the image. At the same time, the sensitivity of the sensor may not be strong, and in a short exposure time, it may not be able to capture most of the stars and the Milky Way nebula clearly, so it is impossible to take a starry sky image with better display effect.
在本公开一些实施例中，可以将网络下载的星图作为先验信息，来辅助星空拍摄。具体的，通过图像的拍摄时间和定位地点，从网络中找到对应的星图，接下来，确定该星图上和拍摄图像对应的区域，进而依据星图区域内星星的灰度调整拍摄图像内星星的灰度。In some embodiments of the present disclosure, a star map downloaded from the network may be used as prior information to assist in shooting a starry sky. Specifically, the corresponding star map is found from the network through the shooting time and location of the image. Next, the area on the star map corresponding to the shot image is determined, and then the image is adjusted according to the grayscale of the stars in the star map area. Grayscale of stars.
为了使与星图匹配准确，需要拍摄的图像达到一定的成像质量。然而，例如手机的终端设备难以达到这种成像质量。另外，此方案需要在网络上寻找对应的星图，完成解析、匹配的操作，过程上不易实现。In order to match the star map accurately, the captured images need to achieve a certain imaging quality. However, it is difficult for terminal devices such as mobile phones to achieve such imaging quality. In addition, this solution needs to find the corresponding star map on the network, and complete the operations of parsing and matching, which is not easy to implement in the process.
此外，鉴于电子设备相机抖动等一些原因，也可能导致拍摄出的图像显示效果不佳。In addition, due to some reasons such as camera shake of electronic equipment, the captured image may not display well.
在这种情况下，本公开示例性实施方式提供了一种新的图像处理方案，以期在一定程度上解决上述问题。Under this circumstance, the exemplary embodiment of the present disclosure provides a new image processing solution to solve the above problems to a certain extent.
图1示出了本公开实施例的图像处理方案的示例性***架构的示意图。FIG. 1 shows a schematic diagram of an exemplary system architecture of an image processing solution according to an embodiment of the present disclosure.
如图1所示，***架构可以包括终端设备1001和服务器1002。终端设备1001与服务器1002可以通过网络连接，网络的连接类型可以例如包括有线、无线通信链路或者光纤 电缆等。As shown in FIG. 1 , the system architecture may include a terminal device 1001 and a server 1002 . The terminal device 1001 and the server 1002 may be connected through a network, and the connection type of the network may include, for example, a wire, a wireless communication link, or an optical fiber cable.
应当理解，终端设备1001和服务器1002的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备和服务器。比如服务器1002可以是多个服务器组成的服务器集群等。服务器1002还可以被称为云端或云端服务器。It should be understood that the numbers of terminal devices 1001 and servers 1002 are merely illustrative. There can be any number of terminal devices and servers according to implementation needs. For example, the server 1002 may be a server cluster composed of multiple servers, or the like. Server 1002 may also be referred to as a cloud or cloud server.
终端设备1001可以通过网络与服务器1002交互，以接收或发送消息等。虽然图1中以智能手机为例示出，然而，终端设备1001还包括平板电脑、智能可穿戴设备、个人计算机等具有拍摄功能的设备。其中，终端设备1001还可以被称为终端、移动终端、移动端、智能终端等。The terminal device 1001 can interact with the server 1002 through the network to receive or send messages and the like. Although a smartphone is shown as an example in FIG. 1 , the terminal device 1001 also includes a tablet computer, a smart wearable device, a personal computer, and other devices with a photographing function. The terminal device 1001 may also be referred to as a terminal, a mobile terminal, a mobile terminal, a smart terminal, or the like.
在由终端设备1001执行本公开示例性实施方式的图像处理过程的情况下，首先，终端设备1001可以对原始图像进行图像分割，确定出原始图像的星空区域和非星空区域。其中，原始图像可以是终端设备1001的相机在连续拍摄夜景的情况下获取到的一帧图像。例如，在一些场景中，可以利用脚架固定终端设备1001，使终端设备1001的相机连续拍摄包含星空的夜景。When the terminal device 1001 performs the image processing process of the exemplary embodiment of the present disclosure, first, the terminal device 1001 can perform image segmentation on the original image to determine the starry sky area and the non-starry sky area of the original image. Wherein, the original image may be a frame of image obtained by the camera of the terminal device 1001 in the case of continuously shooting night scenes. For example, in some scenarios, the terminal device 1001 may be fixed by a tripod, so that the camera of the terminal device 1001 continuously shoots a night scene including a starry sky.
接下来，终端设备1001可以利用参考图像的星空区域，对原始图像的星空区域进行配准处理，得到与原始图像的星空区域对应的第一中间图像。以及，终端设备1001还可以利用参考图像的非星空区域，对原始图像的非星空区域进行配准处理，得到与原始图像的非星空区域对应的第二中间图像。Next, the terminal device 1001 may use the starry sky area of the reference image to perform registration processing on the starry sky area of the original image to obtain a first intermediate image corresponding to the starry sky area of the original image. In addition, the terminal device 1001 may also perform registration processing on the non-starry sky area of the original image by using the non-starry sky area of the reference image to obtain a second intermediate image corresponding to the non-starry sky area of the original image.
随后，终端设备1001可以基于第一中间图像、第二中间图像和原始图像，进行图像融合，以生成第一目标图像。在这种情况下，终端设备1001可以将第一目标图像发送给服务器1002进行保存。Subsequently, the terminal device 1001 may perform image fusion based on the first intermediate image, the second intermediate image and the original image to generate a first target image. In this case, the terminal device 1001 can send the first target image to the server 1002 for saving.
此外，在得到第一目标图像后，终端设备1001还可以对第一目标图像进行星空增强处理，以进一步对图像进行优化。在这种情况下，终端设备1001可以将经历星空增强处理后的第一目标图像发送给服务器1002进行保存。In addition, after obtaining the first target image, the terminal device 1001 may further perform starry sky enhancement processing on the first target image to further optimize the image. In this case, the terminal device 1001 may send the first target image after the starry sky enhancement process to the server 1002 for saving.
在由服务器1002执行本公开示例性实施方式的图像处理过程的情况下，首先，服务器1002可以从终端设备1001获取原始图像，并对原始图像进行图像分割，确定出原始图像的星空区域和非星空区域。此外，原始图像还可以是服务器1002从其他服务器或设备获取到的图像，本公开对此不做限制。When the server 1002 performs the image processing process of the exemplary embodiment of the present disclosure, first, the server 1002 can obtain the original image from the terminal device 1001, and perform image segmentation on the original image to determine the starry sky area and non-starry sky area of the original image area. In addition, the original image may also be an image obtained by the server 1002 from other servers or devices, which is not limited in the present disclosure.
接下来，服务器1002可以利用参考图像的星空区域，对原始图像的星空区域进行配准处理，得到与原始图像的星空区域对应的第一中间图像。以及，服务器1002还可以利用参考图像的非星空区域，对原始图像的非星空区域进行配准处理，得到与原始图像的非星空区域对应的第二中间图像。Next, the server 1002 may perform registration processing on the starry sky area of the original image by using the starry sky area of the reference image to obtain a first intermediate image corresponding to the starry sky area of the original image. In addition, the server 1002 may further perform registration processing on the non-starry sky area of the original image by using the non-starry sky area of the reference image to obtain a second intermediate image corresponding to the non-starry sky area of the original image.
随后，服务器1002可以基于第一中间图像、第二中间图像和原始图像，进行图像融合，以生成第一目标图像。在这种情况下，服务器1002可以将第一目标图像发送给终端设备1001进行展示和/或保存。Subsequently, the server 1002 may perform image fusion based on the first intermediate image, the second intermediate image and the original image to generate a first target image. In this case, the server 1002 may send the first target image to the terminal device 1001 for display and/or storage.
此外，在得到第一目标图像后，服务器1002还可以对第一目标图像进行星空增强处理，以进一步对图像进行优化。在这种情况下，服务器1002可以将经历星空增强处理后的第一目标图像发送给终端设备1001进行展示和/或保存。In addition, after obtaining the first target image, the server 1002 may further perform starry sky enhancement processing on the first target image to further optimize the image. In this case, the server 1002 may send the first target image subjected to the starry sky enhancement process to the terminal device 1001 for display and/or storage.
应当注意的是，上述对图像进行处理的任意步骤均可以由终端设备1001或服务器1002执行，本公开对此不做限制。It should be noted that any step of processing the image above can be performed by the terminal device 1001 or the server 1002, which is not limited in the present disclosure.
图2示出了适于用来实现本公开示例性实施方式的电子设备的示意图。本公开示例性实施方式的终端设备可以被配置为如图2的形式。需要说明的是，图2示出的电子设备仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。Figure 2 shows a schematic diagram of an electronic device suitable for use in implementing exemplary embodiments of the present disclosure. The terminal device of the exemplary embodiment of the present disclosure may be configured in the form of FIG. 2 . It should be noted that the electronic device shown in FIG. 2 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
本公开的电子设备至少包括处理器和存储器，存储器被配置为存储一个或多个程序，当一个或多个程序被处理器执行时，使得处理器可以实现本公开示例性实施方式的图像处理方法。The electronic device of the present disclosure includes at least a processor and a memory, and the memory is configured to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the image processing method of the exemplary embodiment of the present disclosure. .
具体的，如图2所示，电子设备200可以包括：处理器210、内部存储器221、外部存储器接口222、通用串行总线(Universal Serial Bus，USB)接口230、充电管理模块240、电源管理模块241、电池242、天线1、天线2、移动通信模块250、无线通信模块260、音频模块270、扬声器271、受话器272、麦克风273、耳机接口274、传感器模块280、显示屏290、摄像模组291、指示器292、马达293、按键294以及用户标识模块(Subscriber Identification Module，SIM)卡接口295等。其中传感器模块280可以包括深度传感器、压力传感器、陀螺仪传感器、气压传感器、磁传感器、加速度传感器、距离传感器、接近光传感器、指纹传感器、温度传感器、触摸传感器、环境光传感器及骨传导传感器等。Specifically, as shown in FIG. 2, the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, and a power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, headphone jack 274, sensor module 280, display screen 290, camera module 291 , an indicator 292, a motor 293, a key 294, a Subscriber Identification Module (SIM) card interface 295, and the like. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.
可以理解的是，本公开实施例示意的结构并不构成对电子设备200的具体限定。在本公开另一些实施例中，电子设备200可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件、软件或软件和硬件的组合实现。It can be understood that the structures illustrated in the embodiments of the present disclosure do not constitute a specific limitation on the electronic device 200 . In other embodiments of the present disclosure, the electronic device 200 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
处理器210可以包括一个或多个处理单元，例如：处理器210可以包括应用处理器(Application Processor，AP)、调制解调处理器、图形处理器(Graphics Processing Unit，GPU)、图像信号处理器(Image Signal Processor，ISP)、控制器、视频编解码器、数字信号处理器(Digital Signal Processor，DSP)、基带处理器和/或神经网络处理器(Neural-network Processing Unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。另外，处理器210中还可以设置存储器，用于存储指令和数据。The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (Application Processor, AP), a modem processor, a graphics processor (Graphics Processing Unit, GPU), an image signal processor (Image Signal Processor, ISP), controller, video codec, digital signal processor (Digital Signal Processor, DSP), baseband processor and/or Neural-network Processing Unit (NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. In addition, a memory may also be provided in the processor 210 for storing instructions and data.
内部存储器221可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部存储器221可以包括存储程序区和存储数据区。外部存储器接口222可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备200的存储能力。Internal memory 221 may be used to store computer executable program code, which includes instructions. The internal memory 221 may include a storage program area and a storage data area. The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.
电子设备200可以通过ISP、摄像模组291、视频编解码器、GPU、显示屏290及应用处理器等实现拍摄功能。在一些实施例中，电子设备200可以包括1个或N个摄像模组291，N为大于1的正整数，若电子设备200包括N个摄像头，N个摄像头中有一个是主摄像头。The electronic device 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290, the application processor, and the like. In some embodiments, the electronic device 200 may include one or N camera modules 291 , where N is a positive integer greater than 1. If the electronic device 200 includes N cameras, one of the N cameras is the main camera.
本公开方案中提及的原始图像、参考图像以及其他待融合图像中的一个或多个可以由摄像模组291拍摄得到，或者这些图像可以是连续拍摄而得到的连续帧图像。One or more of the original image, reference image, and other images to be fused mentioned in the present disclosure may be captured by the camera module 291, or these images may be continuous frame images obtained by continuous capturing.
例如，图像的曝光方案可以为：曝光时间16秒，ISO300，总计曝光15帧图像，这15帧图像包括上述原始图像、参考图像以及其他待融合图像。需要说明的是，此曝光方案仅是示例性描述，不应作为本公开的限制。For example, the exposure scheme of the image may be: exposure time 16 seconds, ISO 300, a total of 15 frames of images are exposed, and the 15 frames of images include the above-mentioned original image, reference image and other images to be fused. It should be noted that this exposure scheme is only an exemplary description, and should not be used as a limitation of the present disclosure.
另外，需要说明的是，本公开图像处理方案可以应用于多帧短曝的场景中。即使在此场景下，通过本公开的图像配准与融合方案，也可以得到显示效果较好的包含星空的图像，可以呈现低噪清晰的星星和银河星云。In addition, it should be noted that the image processing solution of the present disclosure can be applied to a scene with multiple frames of short exposure. Even in this scenario, through the image registration and fusion scheme of the present disclosure, an image containing a starry sky with better display effect can be obtained, and low-noise and clear stars and galactic nebula can be presented.
本公开还提供了一种计算机可读存储介质，该计算机可读存储介质可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。The present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist alone without being assembled into the electronic device.
计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行***、装置或者器件使用或者与其结合使用。The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
计算机可读存储介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读存储介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。The computer-readable storage medium can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. Program code embodied on a computer-readable storage medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
计算机可读存储介质承载有一个或者多个程序，当上述一个或者多个程序被一个该电 子设备执行时，使得该电子设备实现如下述实施例中所述的方法。The computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to implement the methods described in the following embodiments.
附图中的流程图和框图，图示了按照本公开各种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的***来实现，或者可以用专用硬件与计算机指令的组合来实现。The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现，所描述的单元也可以设置在处理器中。其中，这些单元的名称在某种情况下并不构成对该单元本身的限定。The units involved in the embodiments of the present disclosure may be implemented in software or hardware, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.
下面将以终端设备执行本公开图像处理的过程为例对本公开的图像处理方法进行说明。在这种情况下，下述图像处理装置可以配置在终端设备中。The image processing method of the present disclosure will be described below by taking the process of performing the image processing of the present disclosure by a terminal device as an example. In this case, the image processing apparatus described below may be arranged in the terminal device.
图3示意性示出了本公开的示例性实施方式的图像处理方法的流程图。参考图3，图像处理方法可以包括以下步骤：FIG. 3 schematically shows a flowchart of an image processing method according to an exemplary embodiment of the present disclosure. Referring to Figure 3, the image processing method may include the following steps:
S32.对原始图像进行图像分割，确定出原始图像的星空区域和非星空区域。S32. Perform image segmentation on the original image to determine the starry sky area and the non-starry sky area of the original image.
在本公开的示例性实施方式中，原始图像可以是包含有星空区域的图像。本公开对原始图像的来源不做限制，例如，原始图像可以是终端设备的相机拍摄到的包含星空的图像，具体的，原始图像可以是终端设备的相机在连续拍摄星空夜景的情况下获取到的多帧图像之一，且可以是任意之一。又例如，原始图像可以是终端设备从其他设备上获取到的包含星空的图像。In an exemplary embodiment of the present disclosure, the original image may be an image including a starry sky area. The present disclosure does not limit the source of the original image. For example, the original image may be an image containing a starry sky captured by the camera of the terminal device. Specifically, the original image may be obtained by the camera of the terminal device under the condition of continuously shooting starry night scenes. One of the multi-frame images of , and can be any one. For another example, the original image may be an image containing a starry sky obtained by the terminal device from other devices.
另外，星空区域为包含星星的天空区域。非星空区域为原始图像中除星空区域之外的其他区域。通常，可以将非星空区域认为是地面区域，然而，本公开不限于此，非星空区域还可以包括地面上的房屋、数目、车辆、道路、行人、动物等，甚至非星空区域还可以包括海面、河面等区域。In addition, the sky area is a sky area including stars. Non-sky areas are areas other than star areas in the original image. Generally, the non-starry sky area can be considered as the ground area, however, the present disclosure is not limited to this, the non-starry sky area may also include houses, numbers, vehicles, roads, pedestrians, animals, etc. on the ground, and even the non-starry sky area may also include the sea surface , river and other areas.
根据本公开的一些实施例，可以采用掩膜(mask)的方式对原始图像进行图像分割。也就是说，终端设备可以利用星空区域和非星空区域的掩膜将原始图像分割为星空区域和非星空区域。其中，掩膜的尺寸与原始图像的尺寸相同，掩膜中每一个元素与原始图像上像素坐标一一对应。掩膜中，可以例如用1代表星空区域，用2代表非星空区域。也就是说，可以将掩膜看作一个包含1和2信息的二维矩阵。然而，应当理解的是，1和2仅是示例性的标识，不应作为本公开内容的限制。According to some embodiments of the present disclosure, image segmentation may be performed on the original image by means of a mask. That is to say, the terminal device can use the mask of the starry sky area and the non-sky sky area to divide the original image into the starry sky area and the non-sky sky area. Among them, the size of the mask is the same as the size of the original image, and each element in the mask corresponds to the pixel coordinates on the original image one-to-one. In the mask, for example, 1 can be used to represent the starry sky area, and 2 can be used to represent the non-starry sky area. That is, the mask can be thought of as a two-dimensional matrix containing 1 and 2 information. It should be understood, however, that 1 and 2 are merely exemplary designations and should not be construed as limitations of the present disclosure.
可以获取或预先获取与原始图像的关联图像，并对关联图像进行图像分割，以确定出掩膜。其中，关联图像对应的拍摄场景与原始图像的拍摄场景一致，因此，由关联图像生成的掩膜可以直接应用于原始图像，以确定出原始图像的星空区域和非星空区域。另外，本公开对图像分割的具体过程不做限制，可以利用分割模型来实现，该分割模型可以是深度神经网络，也可以是基于例如Ostu阈值分割等算法形成的模型。The associated image with the original image may be acquired or pre-acquired, and image segmentation may be performed on the associated image to determine the mask. Among them, the shooting scene corresponding to the associated image is consistent with the shooting scene of the original image. Therefore, the mask generated by the associated image can be directly applied to the original image to determine the starry sky area and the non-starry sky area of the original image. In addition, the present disclosure does not limit the specific process of image segmentation, which can be implemented by using a segmentation model. The segmentation model may be a deep neural network, or a model formed based on algorithms such as Ostu threshold segmentation.
关于关联图像的得到过程，一方面，关联图像可以是预先保存的图像，当原始图像的拍摄场景与关联图像的拍摄场景一致时，终端设备可以根据场景直接获取到关联图像。Regarding the process of obtaining the associated image, on the one hand, the associated image may be a pre-saved image. When the shooting scene of the original image is consistent with the shooting scene of the associated image, the terminal device can directly obtain the associated image according to the scene.
另一方面，在终端设备连续拍摄的场景中，首先，可以从连续拍摄的图像中提取若干图像(例如5帧图像、7帧图像等)，并对这多帧图像进行叠加，生成候选关联图像。针对这多帧图像的提取方式，可以是连续帧提取，也可以间隔预定数量帧进行提取，或者随机提取，本公开对此不做限制。接下来，根据候选关联图像，经历格式转换、去噪处理等操作，确定出原始图像的关联图像。应当注意的是，提取出的这多帧图像与原始图像的拍摄 场景相同，在一个实施例中，这多帧图像中还可以包括原始图像。On the other hand, in the scene of continuous shooting by the terminal device, first, several images (such as 5-frame images, 7-frame images, etc.) can be extracted from the continuously shot images, and these multiple frames of images can be superimposed to generate candidate associated images . The extraction method of the multi-frame images may be continuous frame extraction, extraction at intervals of a predetermined number of frames, or random extraction, which is not limited in the present disclosure. Next, according to the candidate associated images, through operations such as format conversion, denoising, etc., the associated images of the original images are determined. It should be noted that the multiple frames of images extracted are the same as the shooting scene of the original image, and in one embodiment, the multiple frames of images may also include the original image.
参考图4对本公开一些实施例得到掩膜的过程进行说明。首先，从连续拍摄的图像数据中获取若干raw格式的图像，对这些图像进行叠加生成候选关联图像。接下来，可以对候选关联图像进行格式转化，得到灰度图，再对灰度图进行去噪处理，得到关联图像。其中，可以例如通过空域滤波的方式进行去噪处理，降低图像中的噪声。随后，可以将关联图像输入分割模型，利用该分割模型输出掩膜。Referring to FIG. 4 , a process of obtaining a mask in some embodiments of the present disclosure will be described. First, several images in raw format are obtained from continuously captured image data, and these images are superimposed to generate candidate associated images. Next, the candidate associated image can be formatted to obtain a grayscale image, and then the grayscale image can be denoised to obtain an associated image. Wherein, denoising processing can be performed, for example, by means of spatial filtering to reduce noise in the image. The associated image can then be fed into a segmentation model, which outputs a mask using the segmentation model.
采用掩膜方式的优点在于，整个图像处理过程仅需要运行一次图像分割过程，在得到掩膜后，可以利用掩膜实现包括原始图像在内的多个图像的星空区域和非星空区域的分割。The advantage of using the mask method is that the entire image processing process only needs to run the image segmentation process once. After the mask is obtained, the mask can be used to achieve the segmentation of the starry sky area and the non-starry sky area of multiple images including the original image.
根据本公开的另一些实施例，也可以每次均利用分割模型对原始图像进行分割处理，以得到对应的星空区域和非星空区域。According to other embodiments of the present disclosure, the original image may also be segmented each time by using the segmentation model to obtain corresponding starry sky areas and non-starry sky areas.
S34.通过参考图像的星空区域和原始图像的星空区域，确定出与原始图像的星空区域对应的第一中间图像。S34. Determine a first intermediate image corresponding to the starry sky area of the original image by referring to the starry sky area of the image and the starry sky area of the original image.
具体的，可以利用参考图像的星空区域，对原始图像的星空区域进行配准处理，得到与原始图像的星空区域对应的第一中间图像。Specifically, the starry sky area of the reference image may be used to perform registration processing on the starry sky area of the original image to obtain a first intermediate image corresponding to the starry sky area of the original image.
根据本公开的一些实施例，参考图像(又可被称为参考帧图像)可以是与原始图像拍摄场景一致的图像，是用于对原始图像进行配准的图像，使图像对齐。在一个实施例中，参考图像可以是上述关联图像，也可以是从网络上下载的图像。According to some embodiments of the present disclosure, the reference image (also referred to as a reference frame image) may be an image consistent with the original image shooting scene, and is an image used for registering the original image to align the images. In one embodiment, the reference image may be the above-mentioned associated image, or may be an image downloaded from the network.
根据本公开的另一些实施例，针对终端设备连续拍摄的场景，参考图像可以是连续拍摄的多帧图像中除原始图像外另一图像。例如，参考图像可以是连续拍摄的多帧图像中除原始图像外的任一图像；又例如，参考图像可以是连续拍摄多帧图像中按拍摄顺序处于中间的图像；再例如，参考图像可以是连续拍摄多帧图像中亮度最大或清晰度最高的图像、本公开对参考图像不做限制。According to other embodiments of the present disclosure, for a scene continuously shot by a terminal device, the reference image may be another image other than the original image among the multiple frames of images shot continuously. For example, the reference image may be any image except the original image among the consecutively shot multiple frames of images; for another example, the reference image may be an image in the middle of the multiple consecutively shot images in the shooting order; for another example, the reference image may be The image with the highest brightness or the highest definition among the multi-frame images is continuously shot, and the present disclosure does not limit the reference image.
应当注意的是，参考图像也被分割为星空区域和非星空区域，然而，本公开对参考图像分割操作的执行时间(时机)不做限制，可以是在原始图像进行分割的同时也进行分割，也可以在原始图像分割之前预先已分割完成，还可以在原始图像进行分割之后再进行分割。It should be noted that the reference image is also divided into a starry sky area and a non-starry sky area. However, the present disclosure does not limit the execution time (timing) of the reference image segmentation operation, and the segmentation may be performed while the original image is segmented, The segmentation can also be completed in advance before the original image is segmented, and the segmentation can also be performed after the original image is segmented.
鉴于星空区域纹理特征不显著，采用提取纹理特征的方式应用于星空区域可能效果不佳。考虑到此，本公开示例性实施方式通过星星来实现星空区域的配准。Since the texture features of the starry sky area are not significant, the method of extracting texture features may not be effective in the starry sky area. With this in mind, an exemplary embodiment of the present disclosure achieves registration of starry sky regions through stars.
参考图5对原始图像的星空区域的配准处理过程进行说明。Referring to FIG. 5 , the registration process of the starry sky area of the original image will be described.
在步骤S502中，将原始图像的星空区域中的星星与参考图像的星空区域的星星进行匹配。In step S502, the stars in the starry sky area of the original image are matched with the stars in the starry sky area of the reference image.
针对原始图像的星空区域，可以确定出星星的像素坐标点。For the starry sky area of the original image, the pixel coordinates of the stars can be determined.
首先，终端设备可以利用原始图像的星空区域的像素统计参数值，计算用于对星空区域中星星进行分割的阈值。First, the terminal device can use the pixel statistical parameter values of the starry sky area of the original image to calculate a threshold for segmenting the stars in the starry sky area.
具体的，对于星空区域选取一子区域，例如可以随机选取一子区域。鉴于星星相对于星空而言，占比较小，因此，子区域中是否有星星几乎不影响方案。Specifically, for the starry sky area to select a sub-area, for example, a sub-area may be randomly selected. Since the proportion of stars is relatively small compared to the sky, whether there are stars in the sub-region hardly affects the scheme.
针对该子区域，计算像素统计参数值。其中像素统计参数值可以包括像素的均值mean和标准差sigma。然后可以将mean+3×sigma作为分割星星的阈值。应当注意的是，还可以采用其他像素统计参数值和计算阈值的方式，本公开对此不做限制。For this sub-region, the pixel statistics parameter value is calculated. The pixel statistical parameter value may include the mean and standard deviation sigma of the pixel. Then mean+3×sigma can be used as the threshold to segment the stars. It should be noted that other methods of pixel statistical parameter values and calculation thresholds may also be used, which are not limited in the present disclosure.
接下来，可以利用确定出的分割星星的阈值，执行阈值分割操作，以从原始图像的星空区域中确定出星星的像素坐标点。Next, a threshold segmentation operation may be performed using the determined threshold for segmenting the stars, so as to determine the pixel coordinate points of the stars from the starry sky area of the original image.
另外，可以将星星所占的像素数作为星星的尺寸(或称为大小)，星星所占像素的像素值的和作为星星的亮度。在本公开的一种示例性实施例中，可以利用星星的尺寸和/或星星的亮度对星空区域的星星进行筛选，通过预先配置阈值，剔除较小和/或亮度较低的星星。In addition, the number of pixels occupied by the stars may be used as the size (or size) of the stars, and the sum of the pixel values of the pixels occupied by the stars may be used as the brightness of the stars. In an exemplary embodiment of the present disclosure, the stars in the starry sky area may be screened by the size of the stars and/or the brightness of the stars, and the smaller and/or lower brightness stars are eliminated by preconfigured thresholds.
在确定出星星的像素坐标点后，可以利用像素坐标点确定出星星的位置。After the pixel coordinate point of the star is determined, the position of the star can be determined by using the pixel coordinate point.
根据本公开的一些实施例，针对一个星星，可以对像素坐标点取平均值，作为该星星 的位置。According to some embodiments of the present disclosure, for a star, the pixel coordinate points may be averaged as the position of the star.
根据本公开的另一些实施例，针对一个星星，首先，可以确定各像素坐标点对应的亮度。接下来，可以根据星星的各像素坐标点以及各像素坐标点对应的亮度，确定该星星的位置。具体的，可以将亮度作为权重，对各像素坐标点进行加权处理，并将加权处理的结果作为该星星的位置。According to other embodiments of the present disclosure, for a star, first, the brightness corresponding to each pixel coordinate point may be determined. Next, the position of the star can be determined according to each pixel coordinate point of the star and the brightness corresponding to each pixel coordinate point. Specifically, the brightness can be used as a weight, and weighting processing can be performed on each pixel coordinate point, and the result of the weighting processing can be used as the position of the star.
在得到原始图像的星空区域中星星的位置后，可以基于原始图像的星空区域中星星的位置，从参考图像的星空区域中确定出与原始图像的星空区域中星星对应的星星，构建星星特征点对，以实现将原始图像的星空区域中的星星与参考图像的星空区域的星星匹配。其中，确定参考图像的星空区域中星星的方式与原始图像相似，不再赘述。类似地，对确定参考图像的星空区域中星星的时机也不做限制。After the positions of the stars in the starry sky area of the original image are obtained, the stars corresponding to the stars in the starry sky area of the original image can be determined from the starry sky area of the reference image based on the positions of the stars in the starry sky area of the original image, and the star feature points can be constructed. Yes, to match the stars in the sky area of the original image with the stars in the sky area of the reference image. The manner of determining the stars in the starry sky area of the reference image is similar to that of the original image, and will not be repeated here. Similarly, there is no restriction on the timing of determining the stars in the starry sky region of the reference image.
针对从参考图像的星空区域中确定出与原始图像的星空区域中星星对应的星星的过程。根据本公开的一些实施例，可以从参考图像的星空区域中确定出距离距原始图像中星星最近的星星。例如，原始图像中星星A的位置为(x，y)，那么对应到参考图像中，以参考图像中(x，y)的位置为基准，确定出距此距离最近的星星A’，作为与原始图像中星星A对应的星星。For the process of determining, from the starry sky region of the reference image, the stars corresponding to the stars in the starry sky region of the original image. According to some embodiments of the present disclosure, the star closest to the star in the original image may be determined from the starry sky region of the reference image. For example, if the position of star A in the original image is (x, y), then it corresponds to the reference image, and the position of (x, y) in the reference image is used as the benchmark to determine the closest star A' to this distance, as the The star corresponding to star A in the original image.
根据本公开另一些实施例，首先，终端设备可以确定参考图像的星空区域中与原始图像的星空区域中星星的位置对应的区域中的星星集合。例如，原始图像中星星A的位置为(x，y)，那么对应到参考图像中，该区域可以是以参考图像中(x，y)的位置为圆心，半径为2×dist1的区域。其中，dist1为实验测得的原始图像与参考图像两帧之间可能移动的最大距离。According to other embodiments of the present disclosure, first, the terminal device may determine a set of stars in a region of the starry sky region of the reference image corresponding to the positions of the stars in the starry sky region of the original image. For example, if the position of star A in the original image is (x, y), then corresponding to the reference image, the area can be an area with the position of (x, y) in the reference image as the center and a radius of 2×dist1. Among them, dist1 is the maximum distance that may move between the original image and the reference image measured by the experiment.
可以理解的是，确定出的星星集合即是参考图像中该区域所有存在的星星的集合。It can be understood that the determined set of stars is the set of all existing stars in the region in the reference image.
接下来，可以从星星集合中确定出与原始图像的星空区域中星星对应的星星。具体的，可以基于星星集合中各星星的亮度和/或尺寸确定出一个星星作为与原始图像的星空区域中星星对应的星星。例如，可以从星星集合中确定出亮度最大的星星，作为与原始图像的星空区域中星星对应的星星；又例如，可以从星星集合中确定出尺寸最大的星星，作为与原始图像的星空区域中星星对应的星星；再例如，可以分别为亮度和尺寸配置权重，通过加权的方式得到星星集合中各星星的评分，选取评分最高的星星，作为与原始图像的星空区域中星星对应的星星。Next, the stars corresponding to the stars in the starry sky region of the original image can be determined from the collection of stars. Specifically, one star may be determined as the star corresponding to the star in the starry sky region of the original image based on the brightness and/or size of each star in the star set. For example, the brightest star can be determined from the set of stars as the star corresponding to the star in the starry sky area of the original image; for another example, the star with the largest size can be determined from the star set as the star corresponding to the star in the starry sky area of the original image. The star corresponding to the star; for another example, weights can be configured for brightness and size respectively, and the score of each star in the star set is obtained by weighting, and the star with the highest score is selected as the star corresponding to the star in the starry sky area of the original image.
可以认为的是，原始图像与参考图像对应的星星为同一星星。It can be considered that the stars corresponding to the original image and the reference image are the same star.
如上所述，在确定出原始图像与参考图像对应的星星后，构建星星特征点对，以实现将原始图像的星空区域中的星星与参考图像的星空区域的星星匹配。As described above, after the stars corresponding to the original image and the reference image are determined, star feature point pairs are constructed to match the stars in the starry sky area of the original image with the stars in the starry sky area of the reference image.
在步骤S504中，根据匹配结果得到第一单应矩阵。In step S504, a first homography matrix is obtained according to the matching result.
鉴于星星匹配的结果中可能存在错误匹配、误匹配的情况。在本公开的示例性实施方式中，可以采用随机抽样一致性(RANSAC，Random Sample Consensus)算法对星星特征点对进行处理，剔除错误的特征点对，计算出针对星空区域的单应矩阵，记为第一单应矩阵。In view of the fact that there may be false matches and mismatches in the results of star matching. In the exemplary embodiment of the present disclosure, the Random Sample Consensus (RANSAC, Random Sample Consensus) algorithm can be used to process the star feature point pairs, eliminate the wrong feature point pairs, calculate the homography matrix for the starry sky area, and record is the first homography matrix.
在步骤S506中，利用第一单应矩阵对原始图像的星空区域进行仿射变换，得到第一中间图像。In step S506, use the first homography matrix to perform affine transformation on the starry sky region of the original image to obtain a first intermediate image.
终端设备可以利用第一单应矩阵对原始图像的星空区域进行仿射变换，得到第一中间图像。其中，仿射变换(warp变换)是一种二维坐标到二维坐标之间的线性变换，具体操作包括但不限于平移、缩放、旋转、翻转等。The terminal device may perform affine transformation on the starry sky area of the original image by using the first homography matrix to obtain the first intermediate image. The affine transformation (warp transformation) is a linear transformation from two-dimensional coordinates to two-dimensional coordinates, and specific operations include but are not limited to translation, scaling, rotation, flipping, and the like.
需要说明的是，第一中间图像是对应于原始图像的星空区域的配准后的图像。It should be noted that the first intermediate image is a registered image corresponding to the starry sky region of the original image.
S36.通过参考图像的非星空区域和原始图像的非星空区域，确定出原始图像的非星空区域对应的第二中间图像。S36. Determine a second intermediate image corresponding to the non-starry sky area of the original image by using the non-starry sky area of the reference image and the non-starry sky area of the original image.
具体的，可以利用参考图像的非星空区域，对原始图像的非星空区域进行配准处理， 得到与原始图像的非星空区域对应的第二中间图像。Specifically, the non-starry sky area of the reference image may be used to perform registration processing on the non-starry sky area of the original image to obtain a second intermediate image corresponding to the non-starry sky area of the original image.
在执行针对非星空区域的配准处理之前，可以对原始图像进行亮度增强，线性提高原始图像的亮度，以增强非星空区域的纹理特征。Before performing the registration process for the non-starry sky area, the original image can be enhanced by brightness, and the brightness of the original image can be linearly increased to enhance the texture features of the non-starry sky area.
参考图6对原始图像的非星空区域的配准处理过程进行说明。Referring to FIG. 6 , the registration process of the non-starry sky area of the original image will be described.
在步骤S602中，提取原始图像的非星空区域的角点特征。In step S602, the corner features of the non-starry sky area of the original image are extracted.
在本公开的示例性实施方式中，针对角点特征的提取过程，可以采用不同的特征提取算法，这些特征提取算法可以包括但不限于FAST特征点检测算法、DOG特征点检测算法、Harris特征点检测算法、SIFT特征点检测算法、SURF特征点检测算法等。特征描述子可以包括但不限于BRIEF特征点描述子、BRISK特征点描述子、FREAK特征点描述子等。In the exemplary embodiment of the present disclosure, different feature extraction algorithms may be used for the extraction process of corner features, and these feature extraction algorithms may include but are not limited to FAST feature point detection algorithm, DOG feature point detection algorithm, Harris feature point detection algorithm Detection algorithm, SIFT feature point detection algorithm, SURF feature point detection algorithm, etc. Feature descriptors may include, but are not limited to, BRIEF feature point descriptors, BRISK feature point descriptors, FREAK feature point descriptors, and the like.
具体的，可以将特征提取算法与特征描述子进行组合，以确定出特征提取模式，例如，特征提取模式可以为FAST特征点检测算法和BRIEF特征点描述子，也可以为DOG特征点检测算法和FREAK特征点描述子。Specifically, a feature extraction algorithm can be combined with a feature descriptor to determine a feature extraction mode. For example, the feature extraction mode can be the FAST feature point detection algorithm and the BRIEF feature point descriptor, or the DOG feature point detection algorithm and FREAK Feature point descriptors.
在步骤S604中，将原始图像的非星空区域的角点特征与参考图像的非星空区域的角点特征进行匹配。In step S604, the corner point features of the non-starry sky area of the original image are matched with the corner point features of the non-starry sky area of the reference image.
确定参考图像的非星空区域的角点特征的方式与原始图像相似，不再赘述。类似地，对确定参考图像的非星空区域的角点特征的时机也不做限制。The manner of determining the corner feature of the non-starry sky area of the reference image is similar to that of the original image, and will not be repeated here. Similarly, the timing of determining the corner feature of the non-starry sky area of the reference image is also not limited.
针对角点特征的匹配过程，本公开示例性实施方式可以采用块匹配算法实现，即基于角点特征，通过确定参考图像中与原始图像的图像块对应的图像块的方式，求出角点的位移，实现角点特征的匹配。For the matching process of corner point features, the exemplary embodiment of the present disclosure can be implemented by using a block matching algorithm, that is, based on the corner point features, by determining the image block in the reference image corresponding to the image block of the original image, the corner point is obtained. Displacement to achieve the matching of corner features.
在步骤S606中，根据匹配结果得到第二单应矩阵。In step S606, a second homography matrix is obtained according to the matching result.
类似地，在本公开的示例性实施方式中，可以采用随机抽样一致性估计出针对非星空区域的单应矩阵，作为第二单应矩阵。Similarly, in an exemplary embodiment of the present disclosure, a homography matrix for a non-starry sky area may be estimated by random sampling consistency as the second homography matrix.
在步骤S608中，利用第二单应矩阵对原始图像的非星空区域进行仿射变换，得到第二中间图像。In step S608, use the second homography matrix to perform affine transformation on the non-starry sky area of the original image to obtain a second intermediate image.
终端设备可以利用第二单应矩阵对原始图像的非星空区域进行仿射变换，得到第二中间图像。需要说明的是，第二中间图像是对应于原始图像的非星空区域的配准后的图像。The terminal device may use the second homography matrix to perform affine transformation on the non-starry sky area of the original image to obtain the second intermediate image. It should be noted that the second intermediate image is a registered image corresponding to the non-starry sky area of the original image.
S38.基于第一中间图像、第二中间图像和原始图像，进行图像融合，并生成第一目标图像。S38. Perform image fusion based on the first intermediate image, the second intermediate image and the original image, and generate a first target image.
在融合时，为了避免星空区域与非星空区域之间交界处融合效果差以及图像中可能存在坏点而造成单应矩阵可能出现错误等问题，本公开进一步将分别星空区域和非星空区域划分为图像块，以图像块的方式进行融合。During the fusion, in order to avoid problems such as poor fusion effect at the junction between the starry sky area and the non-starry sky area and possible bad points in the image, which may cause errors in the homography matrix, the present disclosure further divides the starry sky area and the non-starry sky area into Image blocks, which are fused in the form of image blocks.
例如，图像块的尺寸可以为16×16。然而，本公开对图像块的尺寸不做限制。For example, the size of the image block may be 16×16. However, the present disclosure does not limit the size of the image block.
根据本公开的一些实施例，可以以参考图像为基准，将第一中间图像与原始图像的星空区域进行比较，根据比较结果，确定出需要融合的针对星空区域的图像块集合。具体的，可以计算第一中间图像中图像块与参考图像中对应图像块的相似度，并计算原始图像的星空区域对应图像块与参考图像中该图像块的相似度，从第一中间图像和原始图像的星空区域中选出与参考图像对应图像块相似度大的图像块作为待融合的图像块，进行图像融合。According to some embodiments of the present disclosure, the first intermediate image may be compared with the starry sky area of the original image based on the reference image, and a set of image blocks for the starry sky area that needs to be fused is determined according to the comparison result. Specifically, the similarity between the image block in the first intermediate image and the corresponding image block in the reference image can be calculated, and the similarity between the corresponding image block in the starry sky area of the original image and the image block in the reference image can be calculated. In the starry sky area of the original image, an image block with a large similarity to the corresponding image block of the reference image is selected as the image block to be fused, and image fusion is performed.
类似地，可以以参考图像为基准，将第二中间图像与原始图像的非星空区域进行比较，根据比较结果，确定出需要融合的针对非星空区域的图像块集合。具体的，可以计算第二中间图像中图像块与参考图像中对应图像块的相似度，并计算原始图像的非星空区域对应图像块与参考图像中该图像块的相似度，从第二中间图像和原始图像的非星空区域中选出与参考图像对应图像块相似度大的图像块作为待融合的图像块，进行图像融合。Similarly, the second intermediate image can be compared with the non-starry sky area of the original image based on the reference image, and according to the comparison result, a set of image blocks for the non-starry sky area that needs to be fused is determined. Specifically, the similarity between the image block in the second intermediate image and the corresponding image block in the reference image can be calculated, and the similarity between the image block corresponding to the non-starry sky area of the original image and the image block in the reference image can be calculated. From the second intermediate image In the non-starry sky area with the original image, the image block with a large similarity with the image block corresponding to the reference image is selected as the image block to be fused, and image fusion is performed.
根据本公开的另一些实施例，鉴于第一中间图像对应于星空区域，第二中间图像对应于非星空区域，因此，可以将第一中间图像与第二中间图像合并为与原始图像对应的配准图像。在这种情况下，首先，可以从配准图像和原始图像中确定出待融合图像块。According to other embodiments of the present disclosure, since the first intermediate image corresponds to a starry sky area and the second intermediate image corresponds to a non-starry sky area, the first intermediate image and the second intermediate image may be combined into a matching image corresponding to the original image. standard image. In this case, first, the image patch to be fused can be determined from the registered image and the original image.
具体的，配准图像、原始图像和参考图像的图像块分割方式相同。针对每一个图像块位置，计算配准图像的图像块与参考图像的图像块的相似度，作为第一相似度，并计算原始图像的图像块与参考图像的图像块的相似度，作为第二相似度。Specifically, the image blocks of the registration image, the original image and the reference image are divided in the same manner. For each image block position, the similarity between the image block of the registered image and the image block of the reference image is calculated as the first similarity, and the similarity between the image block of the original image and the image block of the reference image is calculated as the second similarity similarity.
如果第一相似度大于第二相似度，则将配准图像的图像块确定为待融合图像块，不使用原始图像的图像块。If the first similarity is greater than the second similarity, the image block of the registered image is determined as the image block to be fused, and the image block of the original image is not used.
如果第一相似度小于第二相似度，则将原始图像的图像块确定为待融合图像块，不使用配准图像的图像块。If the first similarity is smaller than the second similarity, the image block of the original image is determined as the image block to be fused, and the image block of the registered image is not used.
上述计算图像块相似度的方法可以采用SAD(Sum of Absolute Differences，绝对误差和)算法，鉴于图像块尺寸较小，SAD资源消耗小，工程上可以实现较好的处理效果。然而，还可以采用MAD(Mean Absolute Differences，平均绝对差)算法、SSD(Sum of Squared Differences，误差平方和)算法等，本公开对此不做限制。The above method for calculating the similarity of image blocks can use the SAD (Sum of Absolute Differences, sum of absolute errors) algorithm. In view of the small size of the image blocks and the low consumption of SAD resources, a better processing effect can be achieved in engineering. However, MAD (Mean Absolute Differences, mean absolute differences) algorithm, SSD (Sum of Squared Differences, error sum of squares) algorithm, etc. can also be used, which is not limited in the present disclosure.
在本公开的一些实施例中，图像融合仅仅针对的是原始图像的转换，将与原始图像对应的第一中间图像、第二中间图像和原始图像本身进行融合。也就是说，按图像块的位置，对确定出的待融合图像块进行拼接，生成第一目标图像。In some embodiments of the present disclosure, the image fusion is only aimed at the conversion of the original image, and the first intermediate image, the second intermediate image and the original image itself corresponding to the original image are fused. That is, according to the positions of the image blocks, the determined image blocks to be fused are spliced to generate the first target image.
在本公开的另一些实施例中，第一目标图像由多个待融合图像进行图像融合得到，而上述原始图像为这多个待融合图像中的任意一个。可以理解的是，这多个待融合图像可以是终端设备连续拍摄而得到的多个图像。In other embodiments of the present disclosure, the first target image is obtained by image fusion of multiple images to be fused, and the above-mentioned original image is any one of the multiple images to be fused. It can be understood that the multiple images to be fused may be multiple images obtained by continuous shooting by the terminal device.
参考图7，在存在n个待融合图像时，n为大于等于2的正整数，每一个待融合图像均可以执行上述原始图像的处理过程，即相当于存在n个原始图像。针对原始图像1至原始图像n的每一个原始图像，可以对应得到待融合的图像块集合1至图像块集合n，按图像块位置，将待融合图像块进行融合，生成第一目标图像。其中，融合可以指像素值相加。Referring to FIG. 7 , when there are n images to be fused, n is a positive integer greater than or equal to 2, and each image to be fused can perform the above processing process of the original image, that is, there are n original images. For each original image from original image 1 to original image n, image block set 1 to image block set n to be fused can be obtained correspondingly, and the image blocks to be fused are fused according to the position of the image blocks to generate a first target image. Among them, fusion may refer to the addition of pixel values.
在一个实施例中，如图7所示，在融合的过程中，还可以将参考图像的图像块进行融合。In one embodiment, as shown in FIG. 7 , during the fusion process, image blocks of the reference image may also be fused.
此外，针对生成的第一目标图像，本公开还包括对第一目标图像进一步进行图像优化的方案，以增强星空的显示效果。In addition, for the generated first target image, the present disclosure further includes a solution for further performing image optimization on the first target image, so as to enhance the display effect of the starry sky.
首先，终端设备可以统计出第一目标图像的星空区域非星星像素的亮度均值。根据该亮度均值确定出第一亮度增强系数。First, the terminal device can count the average brightness of non-star pixels in the starry sky area of the first target image. The first luminance enhancement coefficient is determined according to the luminance mean value.
具体的，可以预先设定亮度均值的目标值，该目标值可以由开发人员或用户自行设定。用该目标值除以第一目标图像的星空区域非星星像素的亮度均值，得到第一亮度增强系数。Specifically, the target value of the average brightness value can be preset, and the target value can be set by the developer or the user. Divide the target value by the average brightness of non-star pixels in the starry sky area of the first target image to obtain a first brightness enhancement coefficient.
得到第一亮度增强系数后，可以确定出第二亮度增强系数。其中，第一亮度增强系数小于第二亮度增强系数。After the first brightness enhancement coefficient is obtained, the second brightness enhancement coefficient can be determined. Wherein, the first brightness enhancement coefficient is smaller than the second brightness enhancement coefficient.
接下来，可以利用第一亮度增强系数对第一目标图像进行亮度增强处理，得到第二目标图像。并利用第二亮度增强系数对第一目标图像进行亮度增强处理，得到第三目标图像。其中，第二目标图像又可被称为低增益图像，第三目标图像又可被称为高增益图像。Next, brightness enhancement processing may be performed on the first target image by using the first brightness enhancement coefficient to obtain a second target image. and performing brightness enhancement processing on the first target image by using the second brightness enhancement coefficient to obtain a third target image. The second target image may also be referred to as a low-gain image, and the third target image may also be referred to as a high-gain image.
在本公开一些实施例中，第三目标图像亮度均值是第二目标图像亮度均值的两倍。也就是说，在这些实施例中，还可以先利用第一亮度增强系数对第一目标图像进行亮度增强处理，得到第二目标图像，再对第二目标图像进行线性提亮操作，以生成亮度均值为第二目标图像两倍的第三目标图像。In some embodiments of the present disclosure, the average luminance of the third target image is twice the average luminance of the second target image. That is to say, in these embodiments, the first target image may also be subjected to brightness enhancement processing by using the first brightness enhancement coefficient to obtain a second target image, and then a linear brightening operation may be performed on the second target image to generate brightness A third target image with twice the mean value of the second target image.
然而，需要说明的是，第三目标图像亮度均值是第二目标图像亮度均值的两倍仅是示例性的描述，第三目标图像亮度均值大于第二目标图像亮度均值即可。However, it should be noted that the average brightness of the third target image is twice the average brightness of the second target image is only an exemplary description, and the average brightness of the third target image may be greater than the average brightness of the second target image.
然后，终端设备可以将第二目标图像与第三目标图像融合，以生成第四目标图像。Then, the terminal device may fuse the second target image with the third target image to generate a fourth target image.
针对融合的过程，可以采用图像金字塔融合的方法融合第二目标图像和第三目标图像。For the fusion process, the image pyramid fusion method can be used to fuse the second target image and the third target image.
具体的，对于星空区域中非星星的像素，融合时像素值主要从第二目标图像中提取。对于星空区域中星星的像素，融合时像素值主要从第三目标图像中提取。Specifically, for the pixels that are not stars in the starry sky area, the pixel values are mainly extracted from the second target image during fusion. For the pixels of stars in the starry sky area, the pixel values are mainly extracted from the third target image during fusion.
针对星空区域，本公开还提供了一种第二目标图像和第三目标图像在融合时的权值确 定方案。For the starry sky area, the present disclosure also provides a weight determination scheme when the second target image and the third target image are fused.
一方面，计算第二目标图像中星空区域非星星像素的像素统计参数值，并根据第二目标图像中星空区域非星星像素的像素统计参数值确定第一融合权值。其中，第一融合权值是针对第二目标图像的权值。On the one hand, the pixel statistical parameter values of the non-star pixels in the starry sky area in the second target image are calculated, and the first fusion weight is determined according to the pixel statistical parameter values of the non-star pixels in the starry sky area in the second target image. Wherein, the first fusion weight is a weight for the second target image.
在本公开的一些实施例中，第二目标图像的像素统计参数值对应的是第二目标图像中星空区域非星星像素的统计直方图的波峰。第一融合权值可以被构造成一个高斯权值曲线，该高斯权值曲线要求完全覆盖或尽量完全覆盖该统计直方图的波峰。In some embodiments of the present disclosure, the pixel statistical parameter value of the second target image corresponds to a peak of a statistical histogram of non-star pixels in the starry sky area in the second target image. The first fusion weight can be constructed as a Gaussian weight curve, the Gaussian weight curve is required to completely cover or try to completely cover the peak of the statistical histogram.
另一方面，计算第三目标图像中星空区域非星星像素的像素统计参数值，并根据第三目标图像中星空区域非星星像素的像素统计参数值确定第二融合权值。其中，第二融合权值是针对第三目标图像的权值。On the other hand, the pixel statistical parameter values of the non-star pixels in the starry sky area in the third target image are calculated, and the second fusion weight is determined according to the pixel statistical parameter values of the non-star pixels in the starry sky area in the third target image. Wherein, the second fusion weight is a weight for the third target image.
在本公开的一些实施例中，第三目标图像的像素统计参数值对应的是第三目标图像中星空区域非星星像素的统计直方图的波峰。第二融合权值可以被构造成另一个高斯权值曲线，该高斯权值曲线的波峰要求错开或尽量错开统计直方图的波峰。In some embodiments of the present disclosure, the pixel statistical parameter value of the third target image corresponds to a peak of a statistical histogram of non-star pixels in the starry sky area in the third target image. The second fusion weight may be constructed as another Gaussian weight curve, and the peaks of the Gaussian weight curve are required to be staggered or staggered as far as possible from the peaks of the statistical histogram.
图8示出了本公开实施例的第一融合权值和第二融合权值的示意图。FIG. 8 shows a schematic diagram of a first fusion weight and a second fusion weight in an embodiment of the present disclosure.
参考图8，统计直方图用像素占比表示，第二目标图像中星空区域非星星像素的统计直方图曲线为曲线1，第三目标图像中星空区域非星星像素的统计直方图曲线为曲线2。在这种情况下，本公开实施例的第一融合权值的曲线可以被构造为曲线3，第二融合权值的曲线可以被构造为曲线4。Referring to Figure 8, the statistical histogram is represented by the proportion of pixels, the statistical histogram curve of the non-star pixels in the starry sky area in the second target image is curve 1, and the statistical histogram curve of the non-star pixels in the starry sky area in the third target image is curve 2. . In this case, the curve of the first fusion weight in the embodiment of the present disclosure may be constructed as the curve 3 , and the curve of the second fusion weight may be constructed as the curve 4 .
下面将参考图9对本公开一个实施例的图像处理方法的整个过程进行说明。The entire process of the image processing method according to an embodiment of the present disclosure will be described below with reference to FIG. 9 .
在步骤S902中，终端设备获取连续曝光的多帧raw图像。In step S902 , the terminal device acquires continuously exposed multiple frames of raw images.
在步骤S904中，终端设备对每一帧图像进行分割，得到每一帧图像对应的星空区域和非星空区域。In step S904, the terminal device divides each frame of image to obtain a starry sky area and a non-starry sky area corresponding to each frame of image.
在步骤S906中，终端设备从这些图像中选取一帧作为参考帧。In step S906, the terminal device selects one frame from these images as a reference frame.
在步骤S908中，终端设备选取一帧图像进行与参考帧的配准操作。In step S908, the terminal device selects a frame of image to perform a registration operation with the reference frame.
在步骤S910中，终端设备将选取的图像的星空区域与参考帧的星空区域进行配准。In step S910, the terminal device registers the starry sky area of the selected image with the starry sky area of the reference frame.
在步骤S912中，终端设备将选取的图像的非星空区域与参考帧的非星空区域进行配准。In step S912, the terminal device registers the non-starry sky area of the selected image with the non-starry sky area of the reference frame.
在步骤S914中，终端设备利用配准计算出的单应矩阵分别对原始图像的星空区域和非星空区域进行仿射变换。In step S914, the terminal device performs affine transformation on the starry sky area and the non-starry sky area of the original image respectively by using the homography matrix calculated by the registration.
在步骤S916中，终端设备采用图像块的方式实现图像融合。In step S916, the terminal device implements image fusion in the form of image blocks.
在步骤S918中，终端设备判断是否处理完所有的图像。如果是，则执行步骤S920；如果不是，则返回步骤S906，继续执行下一图像的配准和融合的处理过程。In step S918, the terminal device determines whether all images have been processed. If yes, go to step S920; if not, go back to step S906, and continue to execute the process of registration and fusion of the next image.
在步骤S920中，对融合后的图像进行星空增强。In step S920, starry sky enhancement is performed on the fused image.
在步骤S922中，输出星空增强的图像，例如，经过图像信号处理，输出.jpg格式的图像。In step S922, the enhanced image of the starry sky is output, for example, an image in .jpg format is output after image signal processing.
通常上述图像处理过程，一方面，可以使用户在不需要摄像专业知识和后期处理的情况下，通过简易脚架固定终端设备，一键即可拍摄出包含星空的图像。另一方面，本方案对星星运动和可能的相机抖动进行了补偿并融合多帧图像，使拍摄的总曝光时间不被限制，且不必采用大ISO来完成曝光，因此，可以大大降低图像的噪声，提高了图像的清晰度。再一方面，本公开实施例中通过星空增强的处理手段，可以提升星星的亮度等级，使图像中整个星空更加璀璨绚烂。Usually, the above image processing process, on the one hand, enables users to take a picture including the starry sky with one key by fixing the terminal device through a simple tripod without the need for professional knowledge of photography and post-processing. On the other hand, this scheme compensates for star motion and possible camera shake and fuses multiple frames of images, so that the total exposure time of shooting is not limited, and it is not necessary to use a large ISO to complete the exposure, so the noise of the image can be greatly reduced , which improves the clarity of the image. On the other hand, through the starry sky enhancement processing method in the embodiment of the present disclosure, the brightness level of the stars can be improved, so that the entire starry sky in the image is more splendid and splendid.
应当注意，尽管在附图中以特定顺序描述了本公开中方法的各个步骤，但是，这并非要求或者暗示必须按照该特定顺序来执行这些步骤，或是必须执行全部所示的步骤才能实现期望的结果。附加的或备选的，可以省略某些步骤，将多个步骤合并为一个步骤执行，以及/或者将一个步骤分解为多个步骤执行等。It should be noted that although the various steps of the methods of the present disclosure are depicted in the figures in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps must be performed to achieve the desired the result of. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution, and the like.
进一步的，本示例实施方式中还提供了一种图像处理装置。Further, an image processing apparatus is also provided in this exemplary embodiment.
图10示意性示出了本公开的示例性实施方式的图像处理装置的方框图。参考图10，根据本公开的示例性实施方式的图像处理装置10可以包括图像分割模块101、第一图像确定模块103、第二图像确定模块105、第一图像生成模块107。FIG. 10 schematically shows a block diagram of an image processing apparatus of an exemplary embodiment of the present disclosure. 10 , the image processing apparatus 10 according to an exemplary embodiment of the present disclosure may include an image segmentation module 101 , a first image determination module 103 , a second image determination module 105 , and a first image generation module 107 .
具体的，图像分割模块101可以被配置为对原始图像进行图像分割，确定出原始图像的星空区域和非星空区域；第一图像确定模块103可以通过参考图像的星空区域和原始图像的星空区域，确定出与原始图像的星空区域对应的第一中间图像；第二图像确定模块105可以通过参考图像的非星空区域和原始图像的非星空区域，确定出原始图像的非星空区域对应的第二中间图像；第一图像生成模块107可以被配置为基于第一中间图像、第二中间图像和原始图像，进行图像融合，并生成第一目标图像。Specifically, the image segmentation module 101 may be configured to perform image segmentation on the original image to determine the starry sky area and the non-starry sky area of the original image; the first image determination module 103 may refer to the starry sky area of the image and the starry sky area of the original image, Determine the first intermediate image corresponding to the starry sky area of the original image; the second image determination module 105 can determine the second intermediate image corresponding to the non-starry sky area of the original image by referring to the non-starry sky area of the original image and the non-starry sky area of the original image image; the first image generation module 107 may be configured to perform image fusion based on the first intermediate image, the second intermediate image and the original image, and generate a first target image.
根据本公开的示例性实施例，第一中间图像和第二中间图像构成与原始图像对应的配准图像。在这种情况下，第一图像生成模块107可以被配置为执行：从配准图像和原始图像中确定出待融合图像块；基于待融合图像块，进行图像融合，以生成第一目标图像。According to an exemplary embodiment of the present disclosure, the first intermediate image and the second intermediate image constitute a registered image corresponding to the original image. In this case, the first image generation module 107 may be configured to perform: determining the image blocks to be fused from the registered image and the original image; and performing image fusion based on the image blocks to be fused to generate the first target image.
根据本公开的示例性实施例，第一图像生成模块107从配准图像和原始图像中确定出待融合图像块的过程可以被配置为执行：针对每一个图像块位置，计算配准图像的图像块与参考图像的图像块的相似度，作为第一相似度，以及计算原始图像的图像块与参考图像的图像块的相似度，作为第二相似度；如果第一相似度大于第二相似度，则将配准图像的图像块确定为待融合图像块；如果第一相似度小于第二相似度，则将原始图像的图像块确定为待融合图像块；其中，配准图像、原始图像和参考图像的图像块分割方式相同。According to an exemplary embodiment of the present disclosure, the process in which the first image generation module 107 determines the image block to be fused from the registered image and the original image may be configured to perform: for each image block position, calculate the image of the registered image The similarity between the block and the image block of the reference image is taken as the first similarity, and the similarity between the image block of the original image and the image block of the reference image is calculated as the second similarity; if the first similarity is greater than the second similarity , the image block of the registered image is determined as the image block to be fused; if the first similarity is less than the second similarity, the image block of the original image is determined as the image block to be fused; wherein, the registered image, the original image and the The image blocks of the reference image are divided in the same way.
根据本公开的示例性实施例，第一目标图像由多个待融合图像进行图像融合而得到，多个待融合图像包括原始图像。在这种情况下，第一图像生成模块107基于待融合图像块进行图像融合以生成第一目标图像的过程可以被配置为执行：按图像块位置，将待融合图像块与除原始图像之外的其余待融合图像对应的待融合图像块进行融合，以生成第一目标图像。According to an exemplary embodiment of the present disclosure, the first target image is obtained by performing image fusion of a plurality of images to be fused, and the plurality of images to be fused includes an original image. In this case, the process in which the first image generation module 107 performs image fusion based on the image blocks to be fused to generate the first target image may be configured to perform: by the position of the image blocks, the image blocks to be fused with other than the original image The image blocks to be fused corresponding to the remaining images to be fused are fused to generate a first target image.
根据本公开的示例性实施例，第一图像生成模块107基于待融合图像块进行图像融合以生成第一目标图像的过程还可以被配置为执行：按图像块位置，对确定出的待融合图像块进行拼接，以生成第一目标图像。According to an exemplary embodiment of the present disclosure, the process in which the first image generation module 107 performs image fusion based on the image blocks to be fused to generate the first target image may also be configured to perform: The blocks are stitched to generate the first target image.
根据本公开的示例性实施例，第一图像确定模块103可以被配置为执行：将原始图像的星空区域中的星星与参考图像的星空区域的星星进行匹配；根据匹配结果得到第一单应矩阵；利用第一单应矩阵对原始图像的星空区域进行仿射变换，得到第一中间图像。According to an exemplary embodiment of the present disclosure, the first image determination module 103 may be configured to perform: match the stars in the starry sky area of the original image with the stars in the starry sky area of the reference image; obtain the first homography matrix according to the matching result ; Use the first homography matrix to perform affine transformation on the starry sky area of the original image to obtain the first intermediate image.
根据本公开的示例性实施例，第一图像确定模块103将原始图像的星空区域中的星星与参考图像的星空区域的星星进行匹配的过程可以被配置为执行：确定原始图像的星空区域中星星的位置；基于原始图像的星空区域中星星的位置，从参考图像的星空区域中确定出与原始图像的星空区域中星星对应的星星，构建星星特征点对。According to an exemplary embodiment of the present disclosure, the process in which the first image determination module 103 matches the stars in the starry sky region of the original image with the stars in the starry sky region of the reference image may be configured to perform: determining the stars in the starry sky region of the original image Based on the position of the stars in the starry sky area of the original image, determine the stars corresponding to the stars in the starry sky area of the original image from the starry sky area of the reference image, and construct the star feature point pair.
根据本公开的示例性实施例，第一图像确定模块103确定原始图像的星空区域中星星的位置的过程可以被配置为执行：确定原始图像的星空区域中星星的各像素坐标点；确定各像素坐标点的亮度；根据原始图像的星空区域中星星的各像素坐标点以及各像素坐标点的亮度，确定原始图像的星空区域中星星的位置。According to an exemplary embodiment of the present disclosure, the process in which the first image determination module 103 determines the position of the star in the starry sky area of the original image may be configured to perform: determining each pixel coordinate point of the star in the starry sky area of the original image; determining each pixel The brightness of the coordinate point; according to each pixel coordinate point of the star in the starry sky area of the original image and the brightness of each pixel coordinate point, the position of the star in the starry sky area of the original image is determined.
根据本公开的示例性实施例，第一图像确定模块103确定原始图像的星空区域中星星的各像素坐标点的过程可以被配置为执行：利用原始图像的星空区域的像素统计参数值，计算用于对星空区域中星星进行分割的阈值；利用对星空区域中星星进行分割的阈值，执行阈值分割操作，以从原始图像的星空区域中确定出星星的像素坐标点。According to an exemplary embodiment of the present disclosure, the process in which the first image determination module 103 determines each pixel coordinate point of a star in the starry sky area of the original image may be configured to perform: using the pixel statistical parameter values of the starry sky area of the original image, calculating a The threshold for segmenting the stars in the starry sky area; using the threshold for dividing the stars in the starry sky area, the threshold segmentation operation is performed to determine the pixel coordinate points of the stars from the starry sky area of the original image.
根据本公开的示例性实施例，第一图像确定模块103确定对应的星星的过程可以被配置为执行：确定参考图像的星空区域中与原始图像的星空区域中星星的位置对应的区域中的星星集合；从星星集合中确定出与原始图像的星空区域中星星对应的星星。According to an exemplary embodiment of the present disclosure, the process of determining the corresponding stars by the first image determination module 103 may be configured to perform: determining stars in a region of the starry sky region of the reference image corresponding to the positions of the stars in the starry sky region of the original image Collection; determines from the collection of stars the stars that correspond to the stars in the starry sky region of the original image.
根据本公开的示例性实施例，第一图像确定模块103从星星集合中确定出与原始图像的星空区域中星星对应的星星的过程可以被配置为执行：基于星星集合中各星星的亮度和/或尺寸，从星星集合中确定出与原始图像的星空区域中星星对应的星星。According to an exemplary embodiment of the present disclosure, the process in which the first image determination module 103 determines the stars corresponding to the stars in the starry sky region of the original image from the set of stars may be configured to perform: based on the brightness of each star in the set of stars and/or or size, from the collection of stars to determine the stars that correspond to the stars in the starry sky region of the original image.
根据本公开的示例性实施例，第一图像确定模块103根据匹配结果得到第一单应矩阵的过程可以被配置为执行：利用随机抽样一致性算法对星星特征点对进行处理，剔除错误的特征点对，计算出第一单应矩阵。According to an exemplary embodiment of the present disclosure, the process in which the first image determination module 103 obtains the first homography matrix according to the matching result may be configured to perform: use a random sampling consistency algorithm to process pairs of star feature points, and eliminate erroneous features point pair, the first homography matrix is calculated.
根据本公开的示例性实施例，第二图像确定模块105可以被配置为执行：提取原始图像的非星空区域的角点特征；将原始图像的非星空区域的角点特征与参考图像的非星空区域的角点特征进行匹配；根据匹配结果得到第二单应矩阵；利用第二单应矩阵对原始图像的非星空区域进行仿射变换，得到第二中间图像。According to an exemplary embodiment of the present disclosure, the second image determination module 105 may be configured to perform: extracting corner features of the non-starry sky area of the original image; The corner features of the region are matched; the second homography matrix is obtained according to the matching result; the non-starry sky region of the original image is affinely transformed by the second homography matrix to obtain a second intermediate image.
根据本公开的示例性实施例，第二图像确定模块105还可以被配置为执行：在提取原始图像的非星空区域的角点特征之前，对原始图像进行亮度增强。According to an exemplary embodiment of the present disclosure, the second image determination module 105 may be further configured to perform: before extracting the corner point features of the non-starry sky area of the original image, brightness enhancement is performed on the original image.
根据本公开的示例性实施例，图像分割模块101可以被配置为执行：利用星空区域和非星空区域的掩膜，将原始图像分割为星空区域和非星空区域。According to an exemplary embodiment of the present disclosure, the image segmentation module 101 may be configured to perform: segmenting an original image into a starry sky area and a non-starry sky area using masks of the starry sky area and the non-starry sky area.
根据本公开的示例性实施例，参考图11，相比于图像处理装置10，图像处理装置11还可以包括掩膜确定模块111。According to an exemplary embodiment of the present disclosure, referring to FIG. 11 , compared to the image processing apparatus 10 , the image processing apparatus 11 may further include a mask determination module 111 .
具体的，掩膜确定模块111可以被配置为执行：对原始图像的关联图像进行图像分割，以确定出掩膜。Specifically, the mask determination module 111 may be configured to perform image segmentation on the associated image of the original image to determine the mask.
根据本公开的示例性实施例，掩膜确定模块111还可以被配置为执行：对多帧图像进行叠加，生成候选关联图像；根据候选关联图像，确定出原始图像的关联图像；其中，多帧图像为连续拍摄时与原始图像的拍摄场景相同的图像。According to an exemplary embodiment of the present disclosure, the mask determining module 111 may be further configured to perform: superimposing multiple frames of images to generate candidate associated images; The image is the same image as the shooting scene of the original image when shot continuously.
根据本公开的示例性实施例，参考图12，相比于图像处理装置10，图像处理装置12还可以包括第二图像生成模块121。According to an exemplary embodiment of the present disclosure, referring to FIG. 12 , compared to the image processing apparatus 10 , the image processing apparatus 12 may further include a second image generation module 121 .
具体的，第二图像生成模块121可以被配置为执行：确定第一亮度增强系数和第二亮度增强系数，第一亮度增强系数小于第二亮度增强系数；利用第一亮度增强系数对第一目标图像进行亮度增强处理，得到第二目标图像；利用第二亮度增强系数对第一目标图像进行亮度增强处理，得到第三目标图像；将第二目标图像与第三目标图像融合，以生成第四目标图像。Specifically, the second image generation module 121 may be configured to perform: determining a first brightness enhancement coefficient and a second brightness enhancement coefficient, where the first brightness enhancement coefficient is smaller than the second brightness enhancement coefficient; The image is subjected to brightness enhancement processing to obtain a second target image; the first target image is subjected to brightness enhancement processing using the second brightness enhancement coefficient to obtain a third target image; the second target image and the third target image are fused to generate a fourth target image. target image.
根据本公开的示例性实施例，第二图像生成模块121生成第四目标图像的过程可以被配置为执行：计算第二目标图像中星空区域非星星像素的像素统计参数值，并根据第二目标图像的像素统计参数值确定第一融合权值；计算第三目标图像中星空区域非星星像素的像素统计参数值，并根据第三目标图像的像素统计参数值确定第二融合权值；利用第一融合权值和第二融合权值，将第二目标图像与第三目标图像融合，以生成第四目标图像。According to an exemplary embodiment of the present disclosure, the process of generating the fourth target image by the second image generation module 121 may be configured to perform: calculating pixel statistical parameter values of non-star pixels in the starry sky area in the second target image, and The pixel statistical parameter value of the image determines the first fusion weight; calculates the pixel statistical parameter value of the non-star pixels in the starry sky area in the third target image, and determines the second fusion weight according to the pixel statistical parameter value of the third target image; A fusion weight and a second fusion weight are used to fuse the second target image with the third target image to generate a fourth target image.
由于本公开实施方式的图像处理装置的各个功能模块与上述方法实施方式中相同，因此在此不再赘述。Since each functional module of the image processing apparatus in the embodiment of the present disclosure is the same as that in the above-mentioned method embodiment, it will not be repeated here.
通过以上的实施方式的描述，本领域的技术人员易于理解，这里描述的示例实施方式可以通过软件实现，也可以通过软件结合必要的硬件的方式来实现。因此，根据本公开实施方式的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中或网络上，包括若干指令以使得一台计算设备(可以是个人计算机、服务器、终端装置、或者网络设备等)执行根据本公开实施方式的方法。From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein may be implemented by software, or may be implemented by software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to cause a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to an embodiment of the present disclosure.
此外，上述附图仅是根据本公开示例性实施例的方法所包括的处理的示意性说明，而不是限制目的。易于理解，上述附图所示的处理并不表明或限制这些处理的时间顺序。另外，也易于理解，这些处理可以是例如在多个模块中同步或异步执行的。In addition, the above-mentioned figures are merely schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, in multiple modules.
应当注意，尽管在上文详细描述中提及了用于动作执行的设备的若干模块或者单元， 但是这种划分并非强制性的。实际上，根据本公开的实施方式，上文描述的两个或更多模块或者单元的特征和功能可以在一个模块或者单元中具体化。反之，上文描述的一个模块或者单元的特征和功能可以进一步划分为由多个模块或者单元来具体化。It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.
本领域技术人员在考虑说明书及实践这里公开的内容后，将容易想到本公开的其他实施例。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由权利要求指出。Other embodiments of the present disclosure will readily suggest themselves to those skilled in the art upon consideration of the specification and practice of what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限。It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (20)
- 一种图像处理方法，包括：An image processing method, comprising:对原始图像进行图像分割，确定出所述原始图像的星空区域和非星空区域；Perform image segmentation on the original image to determine the starry sky area and the non-starry sky area of the original image;通过参考图像的星空区域和所述原始图像的星空区域，确定出与所述原始图像的星空区域对应的第一中间图像；Determine a first intermediate image corresponding to the starry sky area of the original image by referring to the starry sky area of the image and the starry sky area of the original image;通过所述参考图像的非星空区域和所述原始图像的非星空区域，确定出所述原始图像的非星空区域对应的第二中间图像；Determine the second intermediate image corresponding to the non-starry sky area of the original image through the non-starry sky area of the reference image and the non-starry sky area of the original image;基于所述第一中间图像、所述第二中间图像和所述原始图像，进行图像融合，并生成第一目标图像。Based on the first intermediate image, the second intermediate image and the original image, image fusion is performed to generate a first target image.
- 根据权利要求1所述的图像处理方法，其中，所述第一中间图像和所述第二中间图像构成与所述原始图像对应的配准图像；其中，基于所述第一中间图像、所述第二中间图像和所述原始图像，进行图像融合，并生成第一目标图像，包括：The image processing method according to claim 1, wherein the first intermediate image and the second intermediate image constitute a registration image corresponding to the original image; wherein, based on the first intermediate image, the Perform image fusion with the second intermediate image and the original image to generate a first target image, including:从所述配准图像和所述原始图像中确定出待融合图像块；Determine the image block to be fused from the registered image and the original image;基于待融合图像块，进行图像融合，以生成第一目标图像。Based on the image blocks to be fused, image fusion is performed to generate a first target image.
- 根据权利要求2所述的图像处理方法，其中，从所述配准图像和所述原始图像中确定出待融合图像块，包括：The image processing method according to claim 2, wherein determining the image block to be fused from the registered image and the original image, comprising:针对每一个图像块位置，计算所述配准图像的图像块与所述参考图像的图像块的相似度，作为第一相似度，以及计算所述原始图像的图像块与所述参考图像的图像块的相似度，作为第二相似度；For each image block position, calculate the similarity between the image block of the registered image and the image block of the reference image as the first similarity, and calculate the image block of the original image and the image of the reference image The similarity of the block, as the second similarity;如果所述第一相似度大于所述第二相似度，则将所述配准图像的图像块确定为所述待融合图像块；If the first similarity is greater than the second similarity, determining the image block of the registered image as the to-be-fused image block;如果所述第一相似度小于所述第二相似度，则将所述原始图像的图像块确定为所述待融合图像块；If the first similarity is less than the second similarity, determining the image block of the original image as the image block to be fused;其中，所述配准图像、所述原始图像和所述参考图像的图像块分割方式相同。Wherein, the image blocks of the registration image, the original image and the reference image are divided in the same manner.
- 根据权利要求2所述的图像处理方法，其中，所述第一目标图像由多个待融合图像进行图像融合而得到，所述多个待融合图像包括所述原始图像；其中，基于待融合图像块，进行图像融合，以生成第一目标图像，包括：The image processing method according to claim 2, wherein the first target image is obtained by image fusion of a plurality of images to be fused, and the plurality of images to be fused includes the original image; wherein, based on the images to be fused block, perform image fusion to generate the first target image, including:按图像块位置，将所述待融合图像块与除所述原始图像之外的其余待融合图像对应的待融合图像块进行融合，以生成第一目标图像。According to the position of the image block, the image block to be fused and the image blocks to be fused corresponding to other images to be fused except the original image are fused to generate a first target image.
- 根据权利要求2所述的图像处理方法，其中，基于待融合图像块，进行图像融合，以生成第一目标图像，包括：The image processing method according to claim 2, wherein, based on the image blocks to be fused, performing image fusion to generate the first target image, comprising:按图像块位置，对确定出的待融合图像块进行拼接，以生成第一目标图像。According to the position of the image blocks, the determined image blocks to be fused are spliced to generate a first target image.
- 根据权利要求1所述的图像处理方法，其中，通过参考图像的星空区域和所述原始图像的星空区域，确定出与所述原始图像的星空区域对应的第一中间图像，包括：The image processing method according to claim 1, wherein determining the first intermediate image corresponding to the starry sky area of the original image by referring to the starry sky area of the image and the starry sky area of the original image, comprising:将所述原始图像的星空区域中的星星与所述参考图像的星空区域的星星进行匹配；matching the stars in the starry sky area of the original image with the stars in the starry sky area of the reference image;根据匹配结果得到第一单应矩阵；Obtain the first homography matrix according to the matching result;利用所述第一单应矩阵对所述原始图像的星空区域进行仿射变换，得到所述第一中间图像。The first intermediate image is obtained by performing affine transformation on the starry sky area of the original image by using the first homography matrix.
- 根据权利要求6所述的图像处理方法，其中，将所述原始图像的星空区域中的星星与所述参考图像的星空区域的星星进行匹配包括：The image processing method according to claim 6, wherein matching the stars in the starry sky area of the original image with the stars in the starry sky area of the reference image comprises:确定所述原始图像的星空区域中星星的位置；determining the positions of the stars in the starry sky region of the original image;基于所述原始图像的星空区域中星星的位置，从所述参考图像的星空区域中确定出与所述原始图像的星空区域中星星对应的星星，构建星星特征点对。Based on the positions of the stars in the starry sky area of the original image, the stars corresponding to the stars in the starry sky area of the original image are determined from the starry sky area of the reference image, and a star feature point pair is constructed.
- 根据权利要求7所述的图像处理方法，其中，确定所述原始图像的星空区域中星 星的位置包括：The image processing method according to claim 7, wherein determining the positions of stars in the starry sky area of the original image comprises:确定所述原始图像的星空区域中星星的各像素坐标点；Determine each pixel coordinate point of the star in the starry sky area of the original image;确定各所述像素坐标点的亮度；determining the brightness of each of the pixel coordinate points;根据所述原始图像的星空区域中星星的各像素坐标点以及各所述像素坐标点的亮度，确定所述原始图像的星空区域中星星的位置。According to each pixel coordinate point of the star in the starry sky area of the original image and the brightness of each pixel coordinate point, the position of the star in the starry sky area of the original image is determined.
- 根据权利要求8所述的图像处理方法，其中，确定所述原始图像的星空区域中星星的各像素坐标点包括：The image processing method according to claim 8, wherein determining each pixel coordinate point of a star in the starry sky area of the original image comprises:利用所述原始图像的星空区域的像素统计参数值，计算用于对星空区域中星星进行分割的阈值；Utilize the pixel statistical parameter value of the starry sky area of the original image, calculate the threshold value used to segment the stars in the starry sky area;利用所述对星空区域中星星进行分割的阈值，执行阈值分割操作，以从所述原始图像的星空区域中确定出星星的像素坐标点。Using the threshold for segmenting the stars in the starry sky area, a threshold value segmentation operation is performed to determine the pixel coordinate points of the stars from the starry sky area of the original image.
- 根据权利要求7所述的图像处理方法，其中，基于所述原始图像的星空区域中星星的位置，从所述参考图像的星空区域中确定出与所述原始图像的星空区域中星星对应的星星，包括：The image processing method according to claim 7, wherein the stars corresponding to the stars in the starry sky area of the original image are determined from the starry sky area of the reference image based on the positions of the stars in the starry sky area of the original image ,include:确定所述参考图像的星空区域中与所述原始图像的星空区域中星星的位置对应的区域中的星星集合；determining the set of stars in the region corresponding to the positions of the stars in the starry sky region of the reference image in the region corresponding to the positions of the stars in the starry sky region of the original image;从所述星星集合中确定出与所述原始图像的星空区域中星星对应的星星。The stars corresponding to the stars in the starry sky region of the original image are determined from the set of stars.
- 根据权利要求10所述的图像处理方法，其中，从所述星星集合中确定出与所述原始图像的星空区域中星星对应的星星包括：The image processing method according to claim 10, wherein determining the stars corresponding to the stars in the starry sky area of the original image from the set of stars comprises:基于所述星星集合中各星星的亮度和/或尺寸，从所述星星集合中确定出与所述原始图像的星空区域中星星对应的星星。The stars corresponding to the stars in the starry sky region of the original image are determined from the set of stars based on the brightness and/or size of each star in the set of stars.
- 根据权利要求7所述的图像处理方法，其中，根据匹配结果得到第一单应矩阵包括：The image processing method according to claim 7, wherein obtaining the first homography matrix according to the matching result comprises:利用随机抽样一致性算法对所述星星特征点对进行处理，剔除错误的特征点对，计算出所述第一单应矩阵。The star feature point pair is processed by a random sampling consistency algorithm, the wrong feature point pair is eliminated, and the first homography matrix is calculated.
- 根据权利要求1所述的图像处理方法，其中，通过所述参考图像的非星空区域和所述原始图像的非星空区域，确定出所述原始图像的非星空区域对应的第二中间图像，包括：The image processing method according to claim 1, wherein the second intermediate image corresponding to the non-starry sky area of the original image is determined through the non-starry sky area of the reference image and the non-starry sky area of the original image, comprising: :提取所述原始图像的非星空区域的角点特征；extracting corner features of the non-starry sky area of the original image;将所述原始图像的非星空区域的角点特征与所述参考图像的非星空区域的角点特征进行匹配；Matching the corner feature of the non-starry sky area of the original image with the corner feature of the non-starry sky area of the reference image;根据匹配结果得到第二单应矩阵；Obtain the second homography matrix according to the matching result;利用所述第二单应矩阵对所述原始图像的非星空区域进行仿射变换，得到所述第二中间图像。The second intermediate image is obtained by performing affine transformation on the non-starry sky area of the original image by using the second homography matrix.
- 根据权利要求13所述的图像处理方法，其中，在提取所述原始图像的非星空区域的角点特征之前，所述图像处理方法还包括：The image processing method according to claim 13, wherein, before extracting the corner feature of the non-starry sky area of the original image, the image processing method further comprises:对所述原始图像进行亮度增强。Brightness enhancement is performed on the original image.
- 根据权利要求1所述的图像处理方法，其中，对原始图像进行图像分割，确定出所述原始图像的星空区域和非星空区域，包括：The image processing method according to claim 1, wherein, performing image segmentation on the original image to determine the starry sky area and the non-starry sky area of the original image, comprising:对多帧图像进行叠加，生成候选关联图像；其中，所述多帧图像为连续拍摄时与所述原始图像的拍摄场景相同的图像；Superimposing multiple frames of images to generate candidate associated images; wherein, the multiple frames of images are images that are the same as the shooting scene of the original image during continuous shooting;根据所述候选关联图像，确定出所述原始图像的关联图像；determining the associated image of the original image according to the candidate associated image;对所述原始图像的关联图像进行图像分割，以确定出星空区域和非星空区域的掩膜；Perform image segmentation on the associated image of the original image to determine the mask of the starry sky area and the non-starry sky area;利用所述掩膜，将所述原始图像分割为星空区域和非星空区域。Using the mask, the original image is segmented into a starry sky area and a non-sky sky area.
- 根据权利要求1至15中任一项所述的图像处理方法，其中，所述图像处理方法 还包括：The image processing method according to any one of claims 1 to 15, wherein the image processing method further comprises:确定第一亮度增强系数和第二亮度增强系数，所述第一亮度增强系数小于所述第二亮度增强系数；determining a first brightness enhancement coefficient and a second brightness enhancement coefficient, the first brightness enhancement coefficient is smaller than the second brightness enhancement coefficient;利用所述第一亮度增强系数对所述第一目标图像进行亮度增强处理，得到第二目标图像；Using the first brightness enhancement coefficient to perform brightness enhancement processing on the first target image to obtain a second target image;利用所述第二亮度增强系数对所述第一目标图像进行亮度增强处理，得到第三目标图像；Using the second brightness enhancement coefficient to perform brightness enhancement processing on the first target image to obtain a third target image;将所述第二目标图像与所述第三目标图像融合，以生成第四目标图像。The second target image is fused with the third target image to generate a fourth target image.
- 根据权利要求16所述的图像处理方法，其中，将所述第二目标图像与所述第三目标图像融合，以生成第四目标图像，包括：The image processing method according to claim 16, wherein fusing the second target image and the third target image to generate a fourth target image comprises:计算所述第二目标图像中星空区域非星星像素的像素统计参数值，并根据所述第二目标图像中星空区域非星星像素的像素统计参数值确定第一融合权值；Calculate the pixel statistical parameter values of the non-star pixels in the starry sky area in the second target image, and determine the first fusion weight according to the pixel statistical parameter values of the non-star pixels in the starry sky area in the second target image;计算所述第三目标图像中星空区域非星星像素的像素统计参数值，并根据所述第三目标图像中星空区域非星星像素的像素统计参数值确定第二融合权值；Calculate the pixel statistical parameter values of the non-star pixels in the starry sky area in the third target image, and determine the second fusion weight according to the pixel statistical parameter values of the non-star pixels in the starry sky area in the third target image;利用所述第一融合权值和所述第二融合权值，将所述第二目标图像与所述第三目标图像融合，以生成第四目标图像。Using the first fusion weight and the second fusion weight, the second target image and the third target image are fused to generate a fourth target image.
- 一种图像处理装置，包括：An image processing device, comprising:图像分割模块，被配置为对原始图像进行图像分割，确定出所述原始图像的星空区域和非星空区域；an image segmentation module, configured to perform image segmentation on the original image, and determine the starry sky area and the non-starry sky area of the original image;第一图像确定模块，被配置为通过参考图像的星空区域和所述原始图像的星空区域，确定出与所述原始图像的星空区域对应的第一中间图像；a first image determination module configured to determine a first intermediate image corresponding to the starry sky area of the original image by referring to the starry sky area of the image and the starry sky area of the original image;第二图像确定模块，被配置为通过所述参考图像的非星空区域和所述原始图像的非星空区域，确定出所述原始图像的非星空区域对应的第二中间图像；A second image determination module configured to determine a second intermediate image corresponding to the non-starry sky area of the original image by using the non-starry sky area of the reference image and the non-starry sky area of the original image;第一图像生成模块，被配置为基于所述第一中间图像、所述第二中间图像和所述原始图像，进行图像融合，并生成第一目标图像。The first image generation module is configured to perform image fusion based on the first intermediate image, the second intermediate image and the original image, and generate a first target image.
- 一种计算机可读存储介质，其上存储有计算机程序，所述程序被处理器执行时实现如权利要求1至17中任一项所述的图像处理方法。A computer-readable storage medium on which a computer program is stored, which implements the image processing method according to any one of claims 1 to 17 when the program is executed by a processor.
- 一种电子设备，包括：An electronic device comprising:处理器；processor;存储器，被配置为存储一个或多个程序，当所述一个或多个程序被所述处理器执行时，使得所述处理器实现如权利要求1至17中任一项所述的图像处理方法。a memory configured to store one or more programs that, when executed by the processor, cause the processor to implement the image processing method of any one of claims 1 to 17 .
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110120725.4 | 2021-01-28 | ||
| CN202110120725.4A CN112802033B (en) | 2021-01-28 | 2021-01-28 | Image processing method and device, computer readable storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022160857A1 true WO2022160857A1 (en) | 2022-08-04 |
Family
ID=75812571
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2021/129615 WO2022160857A1 (en) | 2021-01-28 | 2021-11-09 | Image processing method and apparatus, and computer-readable storage medium and electronic device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112802033B (en) |
| WO (1) | WO2022160857A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112802033B (en) * | 2021-01-28 | 2024-03-19 | Oppo广东移动通信有限公司 | Image processing method and device, computer readable storage medium and electronic equipment |
| CN113409375B (en) * | 2021-07-30 | 2023-02-24 | 浙江大华技术股份有限公司 | Image processing method, image processing apparatus, and non-volatile storage medium |
| CN114885093B (en) * | 2022-03-24 | 2023-06-16 | 岚图汽车科技有限公司 | Vehicle, display control method and device for star sky image of vehicle |
| CN114821030B (en) * | 2022-04-11 | 2023-04-04 | 苏州振旺光电有限公司 | Planet image processing method, system and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110930329A (en) * | 2019-11-20 | 2020-03-27 | 维沃移动通信有限公司 | Starry sky image processing method and device |
| US20200236299A1 (en) * | 2019-01-17 | 2020-07-23 | Olympus Corporation | Image capturing apparatus, image compositing method, and recording medium having recorded therein image compositing program to be executed by computer of image capturing apparatus |
| CN111654623A (en) * | 2020-05-29 | 2020-09-11 | 维沃移动通信有限公司 | Photographing method and device and electronic equipment |
| CN112802033A (en) * | 2021-01-28 | 2021-05-14 | Oppo广东移动通信有限公司 | Image processing method and device, computer readable storage medium and electronic device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110062160B (en) * | 2019-04-09 | 2021-07-02 | Oppo广东移动通信有限公司 | Image processing method and device |
| CN111028189B (en) * | 2019-12-09 | 2023-06-27 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
-
2021
- 2021-01-28 CN CN202110120725.4A patent/CN112802033B/en active Active
- 2021-11-09 WO PCT/CN2021/129615 patent/WO2022160857A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200236299A1 (en) * | 2019-01-17 | 2020-07-23 | Olympus Corporation | Image capturing apparatus, image compositing method, and recording medium having recorded therein image compositing program to be executed by computer of image capturing apparatus |
| CN110930329A (en) * | 2019-11-20 | 2020-03-27 | 维沃移动通信有限公司 | Starry sky image processing method and device |
| CN111654623A (en) * | 2020-05-29 | 2020-09-11 | 维沃移动通信有限公司 | Photographing method and device and electronic equipment |
| CN112802033A (en) * | 2021-01-28 | 2021-05-14 | Oppo广东移动通信有限公司 | Image processing method and device, computer readable storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112802033B (en) | 2024-03-19 |
| CN112802033A (en) | 2021-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022160857A1 (en) | Image processing method and apparatus, and computer-readable storage medium and electronic device | |
| CN108898567B (en) | Image noise reduction method, device and system | |
| US10540806B2 (en) | Systems and methods for depth-assisted perspective distortion correction | |
| CN109242961B (en) | Face modeling method and device, electronic equipment and computer readable medium | |
| WO2018214365A1 (en) | Image correction method, apparatus, device, and system, camera device, and display device | |
| CN109788189B (en) | Five-dimensional video stabilization device and method for fusing camera and gyroscope | |
| WO2016127478A1 (en) | Image processing method and device, and terminal | |
| CN111444744A (en) | Living body detection method, living body detection device, and storage medium | |
| CN116324878A (en) | Segmentation for image effects | |
| US10515463B2 (en) | Object segmentation in a sequence of color image frames by background image and background depth correction | |
| WO2019174544A1 (en) | Image synthesis method and apparatus, and computer storage medium and electronic device | |
| US11527014B2 (en) | Methods and systems for calibrating surface data capture devices | |
| CN107704798B (en) | Image blurring method and device, computer readable storage medium and computer device | |
| US10122912B2 (en) | Device and method for detecting regions in an image | |
| CN108776822B (en) | Target area detection method, device, terminal and storage medium | |
| CN111788603A (en) | Electronic device and method for correcting an image corrected with a first image processing scheme in an external electronic device with a second image processing scheme | |
| CN112767295A (en) | Image processing method, image processing apparatus, storage medium, and electronic device | |
| WO2023098045A1 (en) | Image alignment method and apparatus, and computer device and storage medium | |
| CN111932587A (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| WO2019202511A1 (en) | Object segmentation in a sequence of color image frames based on adaptive foreground mask upsampling | |
| US20170351932A1 (en) | Method, apparatus and computer program product for blur estimation | |
| CN110365897B (en) | Image correction method and device, electronic equipment and computer readable storage medium | |
| CN113205011B (en) | Image mask determining method and device, storage medium and electronic equipment | |
| US10282633B2 (en) | Cross-asset media analysis and processing | |
| WO2022206679A1 (en) | Image processing method and apparatus, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21922447 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21922447 Country of ref document: EP Kind code of ref document: A1 |