CN114549373A

CN114549373A - HDR image generation method and device, electronic equipment and readable storage medium

Info

Publication number: CN114549373A
Application number: CN202011334707.8A
Authority: CN
Inventors: 张恒
Original assignee: Oneplus Technology Shenzhen Co Ltd
Current assignee: Oneplus Technology Shenzhen Co Ltd
Priority date: 2020-11-24
Filing date: 2020-11-24
Publication date: 2022-05-27

Abstract

The application provides a method and a device for generating an HDR image, electronic equipment and a readable storage medium, and relates to the technical field of image processing. The method comprises the following steps: performing image completion processing on the original LDR image with deletion in the multi-frame original LDR image with low dynamic range to obtain a first LDR image of a plurality of frames; aligning the multi-frame first LDR image to obtain a multi-frame second LDR image; and generating a frame HDR image according to the plurality of frames of second LDR images. Thereby, it is avoided that the quality of the generated HDR image is not good due to the presence of defects in the original LDR image.

Description

HDR image generation method and device, electronic equipment and readable storage medium

Technical Field

The present application relates to the field of image processing, and in particular, to a method and an apparatus for generating an HDR image, an electronic device, and a readable storage medium.

Background

The dynamic range of natural light lumen values spans many orders of magnitude, but most digital camera sensors measure only a small fraction of the range. Therefore, LDR (Low Dynamic Range) images measured by digital camera sensors often contain overexposed or underexposed regions, and the ability of human beings to distinguish details in bright and Low dark scenes is not achieved. To solve this problem, HDR (High Dynamic Range) photography has been proposed in order to generate an image that reflects a wider lumen Range.

For this reason, special hardware devices are specially designed for measuring HDR images, but such devices are often expensive and difficult to put into normal use in large quantities. The method for generating one HDR image based on a plurality of LDR images is low in cost, so the method becomes a main way for obtaining the HDR image, but the HDR image obtained by the method generally has poor image quality. Therefore, how to obtain HDR images with high image quality has become a technical problem that needs to be solved by those skilled in the art.

Disclosure of Invention

The embodiment of the application provides a HDR image generation method and device, an electronic device and a readable storage medium.

The embodiment of the application can be realized as follows:

in a first aspect, an embodiment of the present application provides an HDR image generation method, where the method includes:

performing image completion processing on the original LDR image with deletion in the multi-frame original LDR image with the low dynamic range to obtain a first LDR image of the multi-frame;

aligning the multiple frames of first LDR images to obtain multiple frames of second LDR images;

and generating a frame HDR image according to the plurality of frames of second LDR images.

In an optional embodiment, the generating a frame HDR image from the plurality of frames of second LDR images comprises:

respectively extracting global information and local information from the second LDR images of the multiple frames;

obtaining a guide map from a second LDR image as a reference image in an alignment process, wherein a resolution of the guide map is the same as a resolution of the HDR image;

and generating the HDR image according to the global information, the local information and the guide map.

In an optional embodiment, after obtaining the plurality of frames of second LDR images, the method further includes:

processing the multiple frames of second LDR images obtained through alignment by using a pre-trained alignment refinement network to obtain refined multiple frames of second LDR images;

generating a frame HDR image according to the plurality of frames of second LDR images, comprising:

and generating the HDR image according to the refined multi-frame second LDR image.

In an optional embodiment, the performing image completion processing on an original LDR image where a plurality of frames of original LDR images are missing to obtain a plurality of frames of first LDR images includes:

obtaining an edge detection result of each frame of original LDR image;

and according to the edge detection result of each frame of original LDR image, performing edge completion and content completion on the original LDR image with the deletion to obtain a first LDR image.

In an optional embodiment, the obtaining an edge detection result of each frame of the original LDR image includes:

identifying a moving target of each frame of original LDR image to obtain a target area where the moving target is located;

determining a motion region in each frame of original LDR image according to the target region;

and carrying out edge detection on each motion region in each frame of original LDR image to obtain an edge detection result of each frame of original LDR image.

In an optional embodiment, the determining a motion region in each frame of the original LDR image according to the target region includes:

taking one of the original LDR images as a first reference image;

determining two target regions corresponding to the same moving target in the original LDR image of the plurality of frames and the first reference image aiming at the original LDR images except the first reference image;

under the condition that the position difference of the two target areas is larger than a preset value, judging that the two target areas are the motion areas;

and under the condition that the position difference of the two target areas is not larger than a preset value, judging that the target area belonging to the original LDR image of the frame in the two target areas is not a motion area.

In an alternative embodiment, the determining two target regions corresponding to the same moving target in the original LDR image of the frame and the first reference image includes:

calculating the position difference between each target region in the original LDR image of the frame and each target region in the first reference image;

and determining the minimum position difference in the position differences corresponding to each target region in the frame of original LDR image or the first reference image, and taking the two corresponding target regions of the minimum position difference in the frame of original LDR image and the first reference image as two target regions corresponding to the same moving target.

In an optional embodiment, the aligning the multiple frames of the first LDR images to obtain multiple frames of the second LDR images includes:

taking one frame of the first LDR images in the plurality of frames of the first LDR images as a second reference image, wherein the second reference image is taken as one frame of a second LDR image;

obtaining motion change information corresponding to the first LDR image of each other frame according to the first LDR image of each other frame and the second reference image;

and performing reverse transformation on the other frames of the first LDR images according to the motion change information corresponding to the other frames of the first LDR images respectively to obtain second LDR images corresponding to the other frames of the first LDR images.

In an alternative embodiment, the motion change information comprises optical flow estimates,

the obtaining motion change information corresponding to each of the other frames of the first LDR images according to the other frames of the first LDR images and the second reference image includes:

and carrying out local optical flow estimation according to the first LDR image and the second reference image of each other frame to obtain optical flow estimation results corresponding to the first LDR image of each other frame.

In a second aspect, an embodiment of the present application provides an HDR image generation apparatus, including:

the completion module is used for performing image completion processing on the original LDR image with the deletion in the multi-frame original LDR image to obtain a multi-frame first LDR image;

the alignment module is used for carrying out alignment processing on the multi-frame first LDR image to obtain a multi-frame second LDR image;

and the image generation module is used for generating one frame of HDR image according to the multi-frame second LDR image.

In an alternative embodiment, the image generation module is specifically configured to:

In an alternative embodiment, the apparatus further comprises an alignment finishing module,

the alignment refinement module is used for processing the plurality of frames of second LDR images obtained through alignment processing by using a pre-trained alignment refinement network to obtain a plurality of frames of second LDR images after refinement;

the image generation module is specifically configured to generate the HDR image according to the refined multi-frame second LDR image.

In an optional embodiment, the completion module is specifically configured to:

obtaining an edge detection result of each frame of original LDR image;

In an optional embodiment, the completion module is specifically configured to:

taking one of the original LDR images as a first reference image;

In an optional embodiment, the completion module is specifically configured to:

and determining the minimum position difference in the position differences corresponding to each target region in the frame of original LDR image or the first reference image, and taking the two corresponding target regions of the minimum position difference in the frame of original LDR image and the first reference image as the two target regions corresponding to the same moving target.

In an alternative embodiment, the alignment module is specifically configured to:

obtaining motion change information corresponding to the first LDR images of other frames according to the first LDR images of other frames and the second reference image;

In an optional implementation, the motion variation information includes an optical flow estimation result, and the alignment module is specifically configured to:

In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, where the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the HDR image generation method described in any one of the foregoing embodiments.

In a fourth aspect, the present application provides a readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the HDR image generation method as described in any one of the foregoing embodiments.

According to the HDR image generation method, the HDR image generation device, the electronic device and the readable storage medium, firstly, image completion processing is carried out on original LDR images with missing in a plurality of frames of original LDR images to obtain a plurality of frames of first LDR images; then, aligning the multi-frame first LDR image to obtain a multi-frame second LDR image; and finally, generating a frame of HDR image according to the plurality of frames of second LDR images. Therefore, by performing image completion processing on the original LDR image, the problem that the quality of the generated HDR image is poor due to more defects in the original LDR image can be avoided; the image alignment effect can be better, and the quality of the HDR image is further improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.

Fig. 1 is a block schematic diagram of an electronic device provided in an embodiment of the present application;

FIG. 2 is a flowchart illustrating an HDR image generation method provided by an embodiment of the present application;

FIG. 3 is a schematic flow chart of the steps included in step S110 in FIG. 2;

FIG. 4 is a schematic flow chart of the steps involved in step S111 in FIG. 3;

FIG. 5 is one of the schematic diagrams of a target area provided by an embodiment of the present application;

FIG. 6 is a second schematic diagram of a target area provided in an embodiment of the present application;

FIG. 7 is a diagram illustrating an image completion process provided by an embodiment of the present application;

FIG. 8 is a flowchart illustrating steps included in step S120 of FIG. 2;

fig. 9 is a schematic diagram of obtaining a second LDR image according to an embodiment of the present application;

FIG. 10 is a schematic flow chart of the steps involved in step S130 of FIG. 2;

FIG. 11 is a schematic diagram of obtaining the HDR image through a converged network according to an embodiment of the present application;

FIG. 12 is a second flowchart of an HDR image generating method according to an embodiment of the present application;

FIG. 13 is a schematic diagram of a HDR image generation model provided by an embodiment of the present application;

FIG. 14 is a schematic structural diagram of a target recognition network provided in an embodiment of the present application;

fig. 15 is a schematic structural diagram of an edge completion network according to an embodiment of the present application;

fig. 16 is a block schematic diagram of an HDR image generation apparatus provided by an embodiment of the present application;

fig. 17 is a second block schematic diagram of an HDR image generating apparatus according to the second embodiment of the present application.

Icon: 100-an electronic device; 110-a memory; 120-a processor; 130-a communication unit; 200-HDR image generation means; 210-a completion module; 220-an alignment module; 230-an image generation module; 240-alignment refinement Module.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It is noted that relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.

LDR (Low Dynamic Range) images, which refer to images with a Low Dynamic Range, are usually over-exposed or under-exposed. An HDR (High Dynamic Range) image is an image having a High Dynamic Range, and generally, an image is vivid in color and has a gradation feeling, and overexposure or underexposure does not occur. HDR images may provide more dynamic range and image details than LDR images. Currently, multiple LDR images are generally merged to obtain an HDR image. However, when there is an object moving in a scene corresponding to a plurality of LDR images, that is, the same object has different positions in the plurality of LDR images, the HDR obtained by fusion has a significant ghost phenomenon and a blur defect.

Ghost caused by the alignment deviation of the whole image can be solved through panoramic alignment, but the ghost defect caused by the information loss of the object motion and the saturation is difficult to overcome. Some methods attempt to solve this problem by aligning the LDR images more finely before fusion, but these methods all generate other drawbacks due to estimation errors. Some methods propose that moving objects are not considered in fusion or the moving objects are directly removed from a picture, but robust pixel-level moving object identification is difficult to realize and cannot be realized.

The HDR image is synthesized by a multi-frame LDR image, and when the LDR image itself has a fracture, light leakage, and the like, the fused HDR image also has a fusion defect and a ghost, which results in poor image quality of the HDR image.

Therefore, the inventor proposes an HDR image generation method, an apparatus, an electronic device, and a readable storage medium provided in this embodiment of the application, to perform image completion processing on a multi-frame LDR image, then align the multi-frame LDR image after completion processing, and generate one frame of HDR image based on the aligned multi-frame LDR image. Therefore, by performing image completion processing on the original LDR image, the problem that the quality of the generated HDR image is poor due to the defect in the original LDR image can be avoided; the image alignment effect can be better, and the quality of the HDR image is further improved.

The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Referring to fig. 1, fig. 1 is a block diagram of an electronic device 100 according to an embodiment of the present disclosure. The electronic device 100 may be, but is not limited to, a camera, a smart phone, a personal computer, a laptop computer, etc. As shown in fig. 1, the electronic device 100 may include a memory 110, a processor 120, and a communication unit 130. The elements of the memory 110, the processor 120 and the communication unit 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.

The memory 110 is used for storing programs or data. The Memory 110 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.

The processor 120 is used to read/write data or programs stored in the memory 110 and perform corresponding functions. For example, the HDR image generating apparatus 200 is stored in the memory 110, and the HDR image generating apparatus 200 includes at least one software functional module which can be stored in the memory 110 in the form of software or firmware (firmware). The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the HDR image generation apparatus 200 in the embodiment of the present application, so as to implement the HDR image generation method in the embodiment of the present application.

The communication unit 130 is configured to establish a communication connection between the electronic device 100 and another communication terminal through a network, and to transmit and receive data through the network.

It should be understood that the structure shown in fig. 1 is only a schematic structural diagram of the electronic device 100, and the electronic device 100 may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.

Referring to fig. 2, fig. 2 is a flowchart illustrating an HDR image generation method according to an embodiment of the present application. The method may be applied to the electronic device 100. The following describes a specific flow of the HDR image generation method in detail.

And step S110, performing image completion processing on the original LDR image with the missing in the multi-frame original LDR image to obtain a multi-frame first LDR image.

In this embodiment, a plurality of frames of original LDR images may be obtained first. Optionally, multiple shots may be taken of the same scene by a camera, and the multiple frames of LDR images obtained by the shooting are used as the multiple frames of original LDR images; the multi-frame LDR image sent by other equipment can also be directly used as the multi-frame original LDR image. It is understood that the plurality of frames of original LDR images may be obtained in other manners. The respective corresponding exposure times of the multiple frames of original LDR images can be the same or different, and can be set according to actual requirements. The specific number of the original LDR images can be set according to actual requirements, for example, 3 frames.

And after obtaining the plurality of frames of original LDR images, performing image completion processing on the original LDR images with missing in the plurality of frames of original LDR images to obtain the first LDR image. In the imaging or post-processing process of the image, the situation that some pixel regions of the image have information loss inevitably occurs. The image completion aims to generate the content of the missing pixel region by using the information of other pixels in the image so as to achieve the purpose of completion. When image completion is performed on the original LDR image with the missing, any method may be adopted as long as the content missing in the original LDR image can be completed. And step S120, carrying out alignment processing on the multi-frame first LDR image to obtain a multi-frame second LDR image.

When a motion region exists in a multi-frame LDR image, if the multi-frame LDR image is directly fused, a ghost inevitably exists in the obtained HDR image, so that the quality of the HDR image is poor. Therefore, before the fusion, the alignment processing is performed on the multi-frame first LDR image after the image completion processing, so as to obtain a multi-frame second LDR image. The alignment process described above can align the content in the first LDR images of different frames, thereby avoiding the presence of ghosting in the HDR images due to content misalignment.

Moreover, the alignment processing is performed on the first LDR images of multiple frames, and the first LDR images of multiple frames are obtained by performing the completion processing on the original LDR images, so that the poor alignment effect and the poor synthesis effect caused by the missing of the content of the LDR images during the alignment processing can be avoided.

And step S130, generating a frame of HDR image according to the multi-frame second LDR image.

After obtaining the second LDR images of multiple frames through the alignment process, the LDR images of multiple frames may be synthesized into one HDR image. It is of course understood that a frame HDR image may be generated based on the plurality of frames LDR images in any manner.

According to the embodiment of the application, the original LDR image with the deletion is subjected to image completion processing and then is subjected to alignment processing, and then a frame of HDR image is generated based on the processed LDR image, so that the problem that the subsequent alignment effect is poor due to the deletion of the image can be avoided, the image alignment effect is improved, ghost is restrained, and the quality of the generated HDR image is ensured; meanwhile, the HDR image generated by using the image with more defects can be avoided, so that the image quality of the HDR image is further ensured.

As an alternative implementation, please refer to fig. 3, and fig. 3 is a flowchart illustrating a step included in step S110 in fig. 2. Step S110 may include step S111 and step S112.

And step S111, obtaining an edge detection result of each frame of original LDR image.

And step S112, according to the edge detection result of each frame of original LDR image, performing edge completion and content completion on the original LDR image with the missing part to obtain a first LDR image.

In this embodiment, edge detection may be performed on each frame of the original LDR image to obtain an edge detection result of the object in each frame of the original LDR image. And then according to the edge detection result of the object, performing edge completion and content completion on the object in the original LDR image with the missing information, thereby realizing the completion of the missing information to obtain the first LDR image. It is understood that if there is no missing frame of the original LSR image, the frame of the original LDR image may be directly used as the first LDR image of the frame.

Wherein, the edge detection result of each frame of original LDR image can be obtained at the same time, and then the edge completion and the content completion are carried out at the same time; or the edge detection result of each frame of original LDR image may be obtained in sequence, and after the edge detection result of one frame of original LDR image is obtained, edge completion and content completion are performed on the frame of original LDR image under the condition that the frame of original LDR image is missing. It is understood that the specific execution sequence may be determined according to actual situations, and is not limited herein.

The edge detection method includes the steps of obtaining edge detection results of all objects in an entire frame of original LDR image by performing edge detection on the entire frame of original LDR image, and taking the edge detection results of all objects in the entire frame of original LDR image as the edge detection results of the frame of original LDR image. And under the condition that the edge detection result of the frame of original LDR image shows that the original LDT image has loss, performing image completion on the frame of original LDR image. Therefore, each object in the first LDR image obtained after each frame of image completion processing has no information loss, and the quality of the HDR image generated subsequently is further ensured.

Optionally, the image completion processing may be performed on an original LDR image in which at least one frame of the multiple frames of original LDR images is missing. For example, when a plurality of frames of images in the plurality of frames of original LDR images are all missing, image completion can be performed only on the original LDR image in which one frame is missing, so that not only can subsequent alignment effect and synthesis effect be ensured, but also the generation speed of the HDR image can be ensured; and several frames or all the images with the deficiency can be complemented, thereby further ensuring the subsequent alignment effect and the synthesis effect. The specific processing mode can be determined according to actual requirements.

It is of course understood that the number of first LDR images is the same as the number of original LDR images. If only a part of the original LDR images are subjected to image completion processing, the processed original LDR images can be used as a first LDR image of one frame, and other original LDR images without image completion processing can be directly used as the first LDR images.

In general, an image has information loss, which is caused by the motion of an object in an actual scene, so that the motion region detection can be performed on the multiple frames of original LDR images to determine the motion region in each frame of original LDR image, and then the edge detection result of each frame of original LDR image is rapidly obtained based on the motion region.

Alternatively, in this implementation, the motion region in each frame of the original LDR image may be determined by an optical flow analysis algorithm, a target tracking algorithm, or the like. And then carrying out edge detection on each motion area to obtain an edge detection result of each motion area, and taking the edge detection results of all the motion areas in the same frame of the original LDR image as the edge detection result of the frame of the original LDR image. Therefore, only the edge detection is carried out on the motion region, so that the speed of obtaining the edge detection result of each frame of original LDR image can be accelerated, and simultaneously, the problem that the HDR image is poor in quality due to the fact that more information in the LDR image in the HDR image is lost can be still avoided.

Alternatively, in practical application, after each motion region is determined, the edge detection may be performed on the motion region, and then the edge completion and the content completion may be performed.

Optionally, as an optional implementation manner, please refer to fig. 4, and fig. 4 is a schematic flowchart of the step included in step S111 in fig. 3. Step S111 may include steps S1111 to S1113.

And step S1111, identifying a moving object for each frame of original LDR image to obtain a target area where the moving object is located.

Step S1112, determining a motion region in each frame of the original LDR image according to the target region.

And step S1113, performing edge detection on each motion region in each frame of original LDR image to obtain an edge detection result of each frame of original LDR image.

In this embodiment, the moving object identification may be performed on each frame of the original LDR image, so as to identify the moving object in each frame of the LDR image, and the region where the moving object is located is used as the target region, so as to determine the moving region according to the target region. Thus, the motion region can be determined without complex algorithmic processing.

The specific moving target can be set according to actual conditions, such as pedestrians, vehicles and the like. If the HDR image generation method is applied to a night scene, the moving target can be set according to a moving object existing in the night scene; if the HDR image generation method is mainly used in a day scene, the moving target can be set according to a moving object existing in the day scene; if the HDR image generation method can be applied to both the night scene and the day scene, the setting may be performed according to a moving object existing in the night scene and a moving object in the day scene.

Optionally, the electronic device 100 may store a trained target recognition network in advance, and when a motion region needs to be determined, the target recognition network may be used to perform target recognition on each frame of the original LDR image, so as to determine the motion region in each frame of the original LDR image.

In an optional implementation manner, after the target region is determined, the target region may be directly used as a motion region, that is, the target region where the motion target in one frame of the original LDR image is located is directly used as all motion regions of the frame of the original LDR image, and subsequently, the edge detection result of the frame of the original LDR image may be determined according to all the motion regions of the frame of the original LDR image. Thus, it is not necessary to perform image completion processing on all objects in the original LDR image, while still ensuring the quality of the HDR image.

In another optional implementation manner, one of the frames of original LDR images in the multiple frames of original LDR images is used as a first reference image, and after a target region is determined, two target regions corresponding to the same moving target in the frame of original LDR image and the first reference image may be determined for each frame of original LDR image except the first reference image in the multiple frames of original LDR images. Judging whether the position difference of the two target areas is greater than a preset value or not, and if so, judging that the two target areas are motion areas; and if not, judging that the target area belonging to the original LDR image of the frame in the two target areas is not a motion area.

Optionally, a frame of original LDR image may be randomly selected as the first reference image, or one of the frames of original LDR image in the multiple frames of original LDR images may be determined as the first reference image according to a preset first selection rule, or one of the frames of original LDR image in the multiple frames of original LDR images may be determined as the first reference image according to a specified operation of a user. It will of course be appreciated that the first reference image may be determined in other ways.

After the first reference image is determined, the first reference image and each frame of original LDR images except the first reference image in the multiple frames of original LDR images can be used as a group of images, and then two target regions corresponding to the same moving target in each group of images are determined. For example, if there are multiple frames of original LDR images as a map a, a map B, and a map C, where the map a is used as a first reference image, the map a and the map B may be used as a group of images to determine two target regions corresponding to the same moving object in the maps a and B, that is, one target region is located in the map a, the other target region is located in the map B, and the two target regions correspond to the same moving object (e.g., correspond to the same moving vehicle); and taking the graph A and the graph C as a group of images, and determining two target areas corresponding to the same moving target in the graph A and the graph C.

Optionally, two target regions corresponding to the same moving target in each frame of the original LDR image and the first reference image may be determined through a tracking algorithm or other methods.

As an alternative embodiment, two target regions corresponding to the same moving target in one frame of the original LDR image and the first reference image may be determined as follows: calculating the position difference between each target region in the original LDR image of the frame and each target region in the first reference image; and determining the minimum position difference in the position differences corresponding to each target region in the frame of original LDR image or the first reference image, and taking the two corresponding target regions of the minimum position difference in the frame of original LDR image and the first reference image as the two target regions corresponding to the same moving target. Therefore, two target areas corresponding to the same moving target can be determined without complex algorithm processing.

After two target areas corresponding to the same moving target are determined, the position difference of the two target areas is calculated according to the positions of the two target areas. Wherein the position of the target area can be obtained when performing moving object recognition. If the position difference is larger than the preset value, the moving objects in the two target areas move, and therefore the two target areas are judged to be moving areas. If the position difference is smaller than the preset value, the moving objects in the two target areas are not moved, so that the target area which does not belong to the first reference image in the two target areas is judged not to be the moving area, and whether the target area which belongs to the first reference image in the two target areas is the moving area is not judged.

Referring to fig. 5 and 6, fig. 5 is a first schematic diagram of a target area provided in the present embodiment, and fig. 6 is a second schematic diagram of the target area provided in the present embodiment. The manner in which the motion region is determined by the position difference is described below with reference to fig. 5 and 6.

Assume that the multi-frame LDR image includes a diagram a, a diagram B, and a diagram C, where diagram a is a first reference image. After passing the moving object recognition, determining that: the target regions in panel a are a1, a2, the target regions in panel B are B1, B2, and the target regions in panel C are C1, C2.

Next, the image A and the image B can be used as one set of images, and the image A and the image C can be used as another set of images. Then, for each group of images, the position difference between each target area in one image and each target area in the other image is calculated according to the position of the target area in each frame of image, and the two target areas corresponding to the minimum position difference in the position difference corresponding to any one target area are taken as the two target areas corresponding to the same moving target.

For example, for graph a and graph B, the following can be calculated: the position difference a1 between the target areas a1, B1; the position difference a2 between the target areas a1, B2; the position difference a3 between the target areas a2, B1; the position difference a4 between the target areas a2, B2; when the position difference corresponding to the target area a1 is the smallest between a1 and a2 and a1 is found in the target area a1, the target areas a1 and B1 are defined as two target areas corresponding to the same moving object.

Wherein the position of the target area may be represented by coordinates of a center point of the target area. As shown in fig. 6, the position of the target area a1 in panel a may be represented by center point coordinates (x1, y1), and the position of the target area B1 in panel B may be represented by center point coordinates (x2, y 2). Wherein, a1 can be calculated by the following method: a12 ═ 2+ (y1-y2)2 (x1-x 2).

If the a1 is larger than the preset value, the two target areas A1 and B1 corresponding to the same moving target are both target moving areas.

If the target areas a1 and C1 are determined to be two target areas corresponding to the same moving target in the same manner, and the position difference between the target areas a1 and C1 is not greater than the preset value, it is determined that the target area C1 is not a moving area.

Finally, it may be determined that the target regions a1, B1 are motion regions and the target region C1 is not a motion region.

Referring to fig. 7, fig. 7 is a schematic diagram of an image completion process according to an embodiment of the present disclosure. After the motion region is determined, edge detection, edge interpolation, and content interpolation may be performed on the object in the motion region as shown in fig. 7. Optionally, the electronic device 100 may store a pre-trained edge completion network, and the edge detection, the edge completion, and the content completion may be implemented by the edge completion network.

After obtaining the multiple frames of first LDR images, the multiple frames of first LDR images can be aligned through a global optical flow method, a local optical flow method, a trained Gaussian model or other modes, and multiple frames of second LDR images are obtained.

As an alternative implementation, please refer to fig. 8, and fig. 8 is a flowchart illustrating a step included in step S120 in fig. 2. Step S120 may include steps S121 to S123.

And step S121, using one of the plurality of frames of first LDR images as a second reference image.

Optionally, one frame of the first LDR image may be randomly selected as the second reference image, or one frame of the first LDR image in the multiple frames of the first LDR images may be determined as the second reference image according to a preset second selection rule, or one frame of the first LDR image in the multiple frames of the first LDR images may be determined as the second reference image according to a specified operation of the user. It will of course be appreciated that the second reference image may be determined in other ways. Wherein the second reference image is used as a frame of the second LDR image.

And step S122, obtaining motion change information corresponding to the first LDR image of each other frame according to the first LDR image of each other frame and the second reference image.

And step S123, performing reverse transformation on the other frames of the first LDR images according to the motion change information corresponding to the other frames of the first LDR images respectively to obtain second LDR images corresponding to the other frames of the first LDR images.

After the second reference image is determined, the second reference image and the first LDR images of each frame can be used as a group of images to be analyzed. And then, for each group of images to be analyzed, taking the second reference image in the group of images to be analyzed as a reference, analyzing the motion change conditions of other first LDR images and the second reference image of one frame, namely estimating the change conditions of the same moving object in other first LDR images and the second reference image of the frame, and taking the motion change conditions obtained by analysis as the motion change information corresponding to other first LDR images of the frame. Then, the frame of the first LDR image may be inversely transformed according to the motion change information, so as to obtain a second LDR image corresponding to the frame of the first LDR image.

Optionally, fast local optical flow estimation may be performed on each group of images to be analyzed, and an optical flow estimation result corresponding to a first LDR image that is not a second reference image in the group of images to be analyzed is obtained. Wherein the local optical flow estimation may be for a motion region in each set of images to be analyzed. The optical flow estimation result may be a transformation equation. Then, the first LDR image of the frame may be inversely transformed according to the optical flow estimation result, so as to obtain a second LDR image corresponding to the first LDR image of the frame. This makes it possible to obtain an optical flow estimation result quickly and to increase the generation speed of an HDR image.

Referring to fig. 9, fig. 9 is a schematic diagram of obtaining a second LDR image according to an embodiment of the present application. The electronic device 100 may store a pre-trained image alignment network, where the image alignment network includes an optical flow estimation network and an alignment network, and the optical flow estimation network is used to perform local optical flow estimation to obtain an optical flow estimation result Warping function; and the alignment network is used for obtaining a second LDR image through inverse transformation according to the optical flow estimation result Warping function. For example, images Image1 and Image2 are included, Image1 is a second reference Image, Image1 and Image2 are input to the optical flow estimation network, and an optical flow estimation result Warping function corresponding to Image2 is obtained; and performing reverse transformation on the Image2 by adopting an alignment network according to the optical flow estimation result Warping function corresponding to the Image2 to obtain a second LDR Image Warped Image2 corresponding to the Image 2. Wherein, Downsampling in fig. 9 represents Downsampling in the optical flow estimation network, learned features represent features obtained by Downsampling, and reference represents alignment.

After the alignment process is completed, the HDR image can be obtained through the fusion process. The fusion processing may be performed by performing weighted summation on pixel values of the pixels, or may be performed in other manners, which is not specifically limited herein.

Alternatively, in an implementation manner of this embodiment, the HDR image may be quickly generated in a manner shown in fig. 10. Referring to fig. 10, fig. 10 is a flowchart illustrating steps included in step S130 in fig. 2. Step S130 may include steps S131 to S133.

Step S131, respectively extracting global information and local information from the plurality of frames of second LDR images.

In step S132, a guide map is obtained from the second LDR image as a reference image in the alignment process.

Step S133 generates the HDR image according to the global information, the local information, and the guide map.

In this embodiment, the global information and the local information may be extracted for the multi-frame second LDR image obtained through the alignment processing. The global information may include at least any one of color information, target information (e.g., person subject information, etc.), and the local information may include any one of texture information, edge information, and the like. The specific information of the global information and the local information may be set according to actual requirements, which is only an example. After the global information and the local information are obtained, the global information and the local information can be superposed according to the channel direction to obtain a superposition result. Therefore, the global and local information can be effectively utilized and summarized.

Alternatively, the local information and the global information may be directly extracted from the second LDR images of the plurality of frames. The second LDR images of the plurality of frames may be compressed, for example, the images are compressed from 1024 × 768 to 256 × 256, and the local information and the global information are extracted from the compressed second LDR images of the plurality of frames, so that the amount of computation can be reduced.

And, a guide map is also obtained from the second LDR image as a reference image during the alignment process, i.e., a guide map is obtained from the second reference image. Wherein the resolution of the guide map is the same as the resolution of the HDR image. For example, if the resolution of the HDR image is required to be the same as the resolution of the LDR image used when the HDR image is generated, the second LDR image serving as the reference image in the alignment process may be directly used as the guide map. If the resolution of the HDR image is required to be greater than the resolution of the LDR image used, i.e. a high resolution HDR image is required to be obtained, the second LDR image, which is used as the reference image during the alignment process, may be upsampled and then used as the high resolution guide map based on the upsampled result.

Then, the superposition result is fused with the guide map, so that a frame HDR image is generated. Thus, the calculation amount in generating the HDR image can be reduced, and the HDR image can be generated quickly.

Referring to fig. 11, fig. 11 is a schematic diagram of obtaining the HDR image through a converged network according to an embodiment of the present application. The electronic device 100 may store a pre-trained fusion network, where the fusion network may include a Global information extraction network Global branch, a local information extraction network local branch, and a synthetic network binary filtering net. The Global information extraction network Global branch is used for extracting Global information from the multi-frame second LDR image LDR images, and the local information extraction network Lobal branch is used for extracting local information from the multi-frame second LDR image LDR images. And superposing the global information and the local information in the channel direction to obtain superposition results Features. And obtaining the guide map based on the LDR which is used as a reference object in the alignment processing in the multi-frame second LDR image LDR images. And then, fusing the superposition result Features with the guide map by using the binary filtering net as a synthesis network, so as to obtain a frame of HDR image (i.e. the HDR image in fig. 11).

The second LDR image obtained through the alignment process still has a certain difference in detail from the real target image, and the difference is not favorable for ensuring the quality of the generated HDR image. The alignment effect of the target image and the reference image is better than that of the image after the alignment processing and the reference image. In order to reduce the partial difference, the multi-frame second LDR image obtained by the alignment process may be processed again. Referring to fig. 12, fig. 12 is a second flowchart illustrating an HDR image generation method according to an embodiment of the present application. After step S120, the method may further include step S140.

And step S140, processing the multi-frame second LDR image obtained by the alignment processing by using a pre-trained alignment refinement network to obtain a refined multi-frame second LDR image.

The electronic device 100 may further store a pre-trained alignment refinement network, and process a part of details of each frame of the second LDR image by using the alignment refinement network, so as to obtain a plurality of frames of second LDR images after refinement. It will of course be appreciated that since the purpose of aligning the finishing network is primarily to ensure the alignment effect, alignment being indicative of alignment with an image as a reference image, the image before finishing and after finishing may be the same for a second reference image of the plurality of frames of second LDR images, i.e. the first LDR image as a reference image during the alignment process.

After obtaining the refined multi-frame second LDR image, step S130 may be performed according to the refined multi-frame second LDR image, that is, the HDR image is generated according to the refined multi-frame second LDR image. Alternatively, when the HDR image is generated from the refined multiframe second LDR image, the manner shown in fig. 10 may be adopted, and other manners may also be adopted, which is not specifically limited herein.

Referring to fig. 13, fig. 13 is a schematic composition diagram of an HDR image generation model according to an embodiment of the present application. The HDR image generation model may include a motion region detection network, an edge blending network, an image alignment network, an alignment refinement network, and a fusion network (the alignment refinement network and the fusion network are not shown in fig. 13). The motion region detection network is used for detecting motion regions in the multi-frame original LDR image LDR images. The edge completion network is used for performing edge detection, edge completion and content completion on Moving regions to obtain completed Moving regions, and further obtaining multi-frame first LDR image Boundary filtered LDR images. The image alignment network is used for aligning the multi-frame first LDR image bound filtered LDR images to obtain multi-frame second LDR image served images. And the alignment refinement network is used for performing refinement processing on the multiframe second LDR image transmitted images to obtain a processed multiframe second LDR image. The fusion network is used to generate an HDR image (i.e., the HDR image in fig. 13) from the processed multi-frame second LDR image.

Each network in the HDR generation model described above will be specifically described below.

The motion area detection network comprises a target identification network and a motion area determination network. A target recognition model may be trained in advance according to a moving target to be recognized, and the target recognition model may be used as the target recognition network. And identifying the moving object of each frame of original LDR image by using an object identification network to obtain an identification frame, wherein the area where the identification frame is located is the object area where the moving object is located. And the motion area determining network determines two target areas corresponding to the same motion target according to the determined target areas, and determines that the two target areas are motion areas when the position difference of the two target areas is greater than a preset value, so that the motion area in each frame of the original LDR image can be determined.

The data set may be identified using a pre-trained model of mobilenetv 2-SSD. After the recognition results of all images in the data set are obtained, the recognition results are screened and optimized according to the moving target to be recognized, and then the processed data set is used for continuing training the model to obtain the target recognition model meeting the requirements. The screening is to keep the matching of the identification result and the moving target to be identified, and delete the unmatched result; and optimizing, namely marking the unidentified moving objects in the image. Therefore, the optimized data set comprises an image and a mark, wherein the mark refers to the identification of the area where the moving object is located in the image. In the model training mode, the moving target marking of each image in the data set is not needed manually, and meanwhile, the training speed of the model can be accelerated due to the fact that the pre-trained model is used.

When the model is trained, the data set may be determined according to an actual application scenario, for example, if the application scenario of the target recognition model is at night, the night view image may be used as the data set. Alternatively, images similar to the target recognition model application scenario may be screened out from existing public datasets as datasets. The screened images can be adjusted in terms of Hue (Hue), Saturation (Saturation), brightness (Value) and the like, so that the number of images is increased, and the shooting effect under different illumination in an actual scene can be simulated.

Referring to fig. 14, fig. 14 is a schematic structural diagram of a target identification network according to an embodiment of the present application. The structure of the object recognition network conforms to the original Mobilenetv 2-SSD. The skeleton of the target recognition network uses mobilenetv2, eliminating the final conversion and output layers. The framework can reduce the calculation amount and shorten the reasoning time while extracting useful information. The Detection heads (Detection layers) select the fusion of Feature maps (Feature maps) on 6 different scales, and finally regress the coordinates of the target frame (namely, obtain the coordinates of the frame where the target is located) through Non-Maximum Suppression (Non-Maximum Suppression).

In the above object recognition network, a regression L2 loss and a classification L1 loss are used to construct a loss function. Since the target recognition network is not interested in the class, the regression L2 loss, the classification L1 loss use different weights, and the L2 loss weight is greater than the L1 loss weight. For example, L2 loses 0.8 weight and L1 loses 0.2 weight. The loss function L may be of the form:

wherein N represents the number of samples trained per group; l is_confRepresents a classification loss, i.e., a classified L1 loss; l is_locRepresents the coordinate loss, i.e., the regressive L2 loss; a represents a preset weight of coordinate loss, such as 0.8.

The cascaded convolutional neural network can be used as an edge completion network to complete missing information of a moving object in an image. Referring to fig. 15, fig. 15 is a schematic structural diagram of an edge completion network according to an embodiment of the present disclosure. The edge completion network may include an edge detection network and two generative countermeasure networks, wherein the edge detection network is used in the first stage, and the two generative countermeasure networks are used in the second and third stages, respectively. In the first stage, the edge detection network is used for detecting an Incomplete edge infinished contact of the image Incomplex image, and then the obtained Incomplete edge infinished contact and the image Incomplex image are sent to the first countermeasure generation network. In the second stage, the generation network in the first generation countermeasure network mainly carries out edge completion to obtain a complete edge Finished constraint; the decision network in the first generative countermeasure network is used to resolve whether the generated complete edge Finished contourer meets the requirements. In the third stage, the complete edge Finished constraint and the image Incomplex image are superposed and then sent to a second generated countermeasure network for refinement, the generation network in the second generated countermeasure network is used for generating a complete image Recovered image, and a judgment network in the second generated countermeasure network is used for judging whether the generated result meets the requirement or not.

In training the edge completion network, the edge completion network may be trained using simulation plus real data. The simulation data is formed from a common data set, and for each moving object to be recognized in an image in the common data set, the simulation data is artificially processed so that information is missing in the processed image. For example, the edges of moving objects in the image are eroded and broken. The processed image and the image before processing are used as simulation data for training. And acquiring a real image which comprises a moving target and has information loss, and then performing completion processing on the real image by using a morphological method or other methods, wherein the image before and after the completion processing is used as real data used in training. Combining the simulation data and the real data, the edge completion network can be trained, and the edge completion network can achieve the purpose of completing the default area in the graph.

The image alignment network may include an optical flow estimation network and an alignment network. The optical flow estimation network may adopt a common lightweight optical flow estimation model, such as FlowNetS, for performing optical flow estimation on the reference image and the image to be transformed to obtain an optical flow estimation result corresponding to the image to be transformed. And aligning the network, and performing inverse transformation on the image to be transformed according to the optical flow estimation result so as to obtain an image corresponding to the reference image.

Alternatively, the optical flow estimation network used may be FlowNetS. When optical flow estimation is performed, two frames of images can be directly input after being overlapped according to red (red), green (green) and blue (blue) channels. In the process of decoding details, for the deconvolution ReLU layer of each layer, not only the output of the previous layer is input, but also the predicted low-scale optical flow of the previous layer and the feature layer in the corresponding coding module are input. Therefore, when each deconvolution layer is thinned, deep abstract information can be obtained, and superficial image information can be obtained at the same time, so that information lost due to reduction of the characteristic space dimension is made up.

When the optical flow estimation network is trained, images similar to the actual application scene can be screened out from the public data set to be used as sample images for subsequent training, or the images in the public data set are processed in a memorability mode, and the processed images are used as sample images for subsequent training. For example, if the actual application scene is at night, most of the images in the public data set are the scenes in the day, the style filter can be used for carrying out style conversion on the images, so that a large number of night scene images are generated, the images similar to the actual night scene are screened out from the generated night scene images to serve as subsequent sample images, and the images with larger difference from the actual scene are removed.

The loss function of the optical flow estimation network can use the loss function of a primary optical flow estimation network (namely, the existing optical flow estimation network), but for a motion area, a larger weight can be based, so that the optical flow estimation network can be helped to pay more attention to remarkable motion.

The alignment refinement network can be obtained by training according to the aligned image and the target image corresponding to the aligned image. Alternatively, the alignment refinement network may be an auto-encoder based on a convolutional neural network, the design of which is based on classical Unet. To facilitate that the model can be deployed on most devices (e.g., on a smartphone), the number of convolution channels and the maximum pooling metric can be adjusted accordingly when training to obtain an alignment refinement network. Alternatively, the alignment refinement network may be deployed on most devices in several ways. When the refinement network is trained, the samples used (i.e., the images after alignment processing and the target images corresponding to the images after alignment processing) may be predetermined in any manner.

The alignment refinement network is obtained by using a Neural network Search method, i.e., training based on a Neural Architecture Search (NAS). Basic sub-network modules required for searching can be specified, parameters (such as MACs) and inference time used by target equipment (namely equipment deployed with the alignment refinement network) are matched to serve as search constraints, and a neural network search model is trained, so that a neural network which can be used for performing alignment refinement on pictures is obtained.

The optimization of the network can also be performed using a model pruning method. The pruning method comprises channel pruning, sparse pruning and the like. Optionally, a model pruning method based on batch sparse constraint may be used to obtain an alignment refinement network meeting the requirements.

Optimization of the network can also be performed using a method of model distillation. Firstly, the original Unet input and output size and the parameters of the middle hidden layer can be modified correspondingly, then a private data set is used for large-scale training, and the training result is used as a teacher model of the distillation process. Second, half-channel Unet was designed as a student model. The self-designed characteristic loss function and the weighted sum of sparse constraint regularization can be used as the comprehensive loss function of the training model, the model is trained on a large scale, and finally a small model which is smaller than the original model by 3/4 and has the same alignment precision as the original model is obtained, so that the deployment at the end of the smart phone is possible.

The converged network can be an HDR deep learning model, which can be a lightweight model, but also other models or networks. Alternatively, the structure of the converged network may be as shown in fig. 11, and the converged network may include at least a Global branch, a local branch, and a binary filtering net. The Global information extraction network Global branch, the local information extraction network Lobal branch and the synthesis network binary filtering net can be trained by utilizing the training sample, so that an alignment finishing network meeting the requirement is obtained, the alignment finishing network can be utilized to realize finishing processing in actual use, and the quality of the generated HDR image is further ensured. Each training sample may include a plurality of frames of LDR images (which may be LDR images obtained after alignment processing, or a plurality of frames of second LDR images in this embodiment, or refined plurality of frames of second LDR images in this embodiment), and a sample HDR image corresponding to the plurality of frames of LDR images. Optionally, in the training process, the guide map may be obtained from a frame of LDR image of the multiple frames of LDR images serving as the reference image in the alignment process, or may be obtained from an HDR image of the sample in the current training sample. The format of the image input into the converged network may be, but is not limited to, RGB, YUV, raw, or the like. The network may be trained separately for different data formats for processing.

Optionally, the operation area detection network, the edge completion network, the image alignment network, and the alignment refinement network may be pre-trained to fix the weight of each portion. After the pre-training is completed, the fusion network is connected in series to perform the joint training, that is, the fusion network is mainly trained during the joint training, and parameters of other parts can be adjusted during the joint training, so that the network obtained based on the training can generate a high-quality HDR image. Therefore, the complex problem of ghost image removal can be divided into networks to be processed respectively, and then the networks are connected in series to form a final HDR image generation model.

Each part of the HDR image generation model can adopt a light-weight neural network, so that the method has the characteristic of high processing speed, can synthesize the KDR image in near real time, and is convenient to deploy to a mobile terminal. The HDR image generation model can eliminate the problem of fusion defect caused by the defect of the original LDR image by performing image completion on the motion region; and performing optical flow estimation on the local part of the image after image completion, and converting the image based on the optical flow estimation result, so that the alignment effect of the image is better, and ghost is better inhibited. Moreover, after the alignment processing, the alignment finishing processing is performed on the alignment processing result, so that the image alignment effect is better.

In order to perform the corresponding steps in the above embodiments and various possible manners, an implementation manner of the HDR image generation apparatus 200 is given below, and optionally, the HDR image generation apparatus 200 may adopt the device structure of the electronic device 100 shown in fig. 1. Further, referring to fig. 16, fig. 16 is a block diagram illustrating an HDR image generating apparatus 200 according to an embodiment of the present application. It should be noted that the basic principle and the technical effect of the HDR image generating apparatus 200 provided in the present embodiment are the same as those of the above embodiments, and for the sake of brief description, no part of the present embodiment is mentioned, and corresponding contents in the above embodiments may be referred to. The HDR image generation apparatus 200 is applied to the electronic device 100. The HDR image generation apparatus 200 may include a completion module 210, an alignment module 220, and an image generation module 230.

The completion module 210 is configured to perform image completion processing on an original LDR image with a missing in multiple frames of the original LDR image, so as to obtain a multiple frames of the first LDR image.

The alignment module 220 is configured to perform alignment processing on the multiple frames of the first LDR image to obtain multiple frames of the second LDR image.

The image generating module 230 is configured to generate a frame HDR image according to the multiple frames of second LDR images.

In an alternative embodiment, the image generating module 230 is specifically configured to: respectively extracting global information and local information from the second LDR images of the multiple frames; obtaining a guide map from a second LDR image as a reference image in an alignment process, wherein a resolution of the guide map is the same as a resolution of the HDR image; and generating the HDR image according to the global information, the local information and the guide map.

In an alternative embodiment, please refer to fig. 17, fig. 17 is a second block diagram of an HDR image generating apparatus 200 according to an embodiment of the present application. The HDR image generation apparatus 200 may further comprise an alignment refinement module 240. The alignment refinement module 240 is configured to process the multiple frames of second LDR images obtained through alignment processing by using a pre-trained alignment refinement network, so as to obtain refined multiple frames of second LDR images. The image generating module 230 is specifically configured to generate the HDR image according to the refined multi-frame second LDR image.

In an optional embodiment, the completion module 210 is specifically configured to: obtaining an edge detection result of each frame of original LDR image; and according to the edge detection result of each frame of original LDR image, performing edge completion and content completion on the original LDR image with the deletion to obtain a first LDR image.

In an optional embodiment, the completion module 210 is specifically configured to: identifying a moving target of each frame of original LDR image to obtain a target area where the moving target is located; determining a motion region in each frame of original LDR image according to the target region; and carrying out edge detection on each motion region in each frame of original LDR image to obtain an edge detection result of each frame of original LDR image.

In an optional embodiment, the completion module 210 is specifically configured to: taking one of the original LDR images of the plurality of frames of original LDR images as a first reference image; determining two target regions corresponding to the same moving target in the original LDR image of the plurality of frames and the first reference image aiming at the original LDR images except the first reference image; under the condition that the position difference of the two target areas is larger than a preset value, judging that the two target areas are the motion areas; and under the condition that the position difference of the two target areas is not larger than a preset value, judging that the target area belonging to the original LDR image of the frame in the two target areas is not a motion area.

In an optional embodiment, the completion module 210 is specifically configured to: calculating the position difference between each target region in the original LDR image of the frame and each target region in the first reference image; and determining the minimum position difference in the position differences corresponding to each target region in the frame of original LDR image or the first reference image, and taking the two corresponding target regions of the minimum position difference in the frame of original LDR image and the first reference image as the two target regions corresponding to the same moving target.

In an alternative embodiment, the alignment module 220 is specifically configured to: taking one frame of the first LDR images in the plurality of frames of the first LDR images as a second reference image, wherein the second reference image is taken as one frame of a second LDR image; obtaining motion change information corresponding to the first LDR images of other frames according to the first LDR images of other frames and the second reference image; and performing reverse transformation on the other frames of the first LDR images according to the motion change information corresponding to the other frames of the first LDR images respectively to obtain second LDR images corresponding to the other frames of the first LDR images.

In an optional implementation, the operation variation information includes an optical flow estimation result, and the alignment module 220 is specifically configured to: and carrying out local optical flow estimation according to the first LDR image and the second reference image of each other frame to obtain optical flow estimation results corresponding to the first LDR image of each other frame.

Alternatively, as an alternative embodiment, the HDR image generation model shown in fig. 13 may be used as the HDR image generation apparatus 200.

Embodiments of the present application further provide a readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the HDR image generation method.

To sum up, the embodiment of the present application provides a method, an apparatus, an electronic device, and a readable storage medium for generating an HDR image, where an original LDR image with a deletion in multiple frames of original LDR images is first subjected to image completion processing to obtain a multiple frames of first LDR images; then, aligning the multi-frame first LDR image to obtain a multi-frame second LDR image; and finally, generating a frame of HDR image according to the plurality of frames of second LDR images. Therefore, by performing image completion processing on the original LDR image, the problem that the quality of the generated HDR image is poor due to more defects in the original LDR image can be avoided; the image alignment effect can be better, and the quality of the HDR image is further improved.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of high dynamic range, HDR, image generation, the method comprising:

performing image completion processing on the original LDR image with deletion in the multi-frame original LDR image with low dynamic range to obtain a first LDR image of a plurality of frames;

2. The method of claim 1, wherein generating a HDR image from the second LDR images comprises:

3. The method of claim 1,

after obtaining the plurality of frames of second LDR images, the method further includes:

4. The method according to any one of claims 1 to 3, wherein the performing image completion processing on the original LDR image with the missing in the plurality of frames of original LDR images to obtain a plurality of frames of first LDR images comprises:

obtaining an edge detection result of each frame of original LDR image;

5. The method of claim 4, wherein obtaining the edge detection result of each frame of the original LDR image comprises:

identifying a moving object of each frame of original LDR image to obtain a target area where the moving object is located;

6. The method of claim 5, wherein determining the motion region in each frame of the original LDR image according to the target region comprises:

taking one of the original LDR images as a first reference image;

7. The method of claim 6, wherein said determining two target regions corresponding to the same moving object in the original LDR image and the first reference image comprises:

8. The method according to any one of claims 1 to 3, wherein the aligning the plurality of frames of the first LDR images to obtain a plurality of frames of the second LDR images comprises:

9. The method of claim 8, wherein the motion change information comprises optical flow estimates,

10. An HDR image generation apparatus, characterized in that the apparatus comprises:

11. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the HDR image generation method of any of claims 1-9.

12. A readable storage medium on which a computer program is stored, which computer program, when being executed by a processor, carries out the HDR image generation method of any one of claims 1 to 9.