WO2021135676A1 - Photographing background blurring method, mobile terminal, and storage medium - Google Patents
Photographing background blurring method, mobile terminal, and storage medium Download PDFInfo
- Publication number
- WO2021135676A1 WO2021135676A1 PCT/CN2020/128657 CN2020128657W WO2021135676A1 WO 2021135676 A1 WO2021135676 A1 WO 2021135676A1 CN 2020128657 W CN2020128657 W CN 2020128657W WO 2021135676 A1 WO2021135676 A1 WO 2021135676A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- background
- picture
- foreground
- blurring
- blur
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 101
- 238000007781 pre-processing Methods 0.000 claims abstract description 23
- 230000007704 transition Effects 0.000 claims description 43
- 230000011218 segmentation Effects 0.000 claims description 39
- 238000001914 filtration Methods 0.000 claims description 31
- 230000009466 transformation Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000003628 erosive effect Effects 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 18
- 230000008569 process Effects 0.000 description 7
- 125000001475 halogen functional group Chemical group 0.000 description 5
- 238000009499 grossing Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000009977 dual effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000012512 characterization method Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005530 etching Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
Definitions
- the present disclosure relates to the technical field of image processing, and in particular to a method for blurring the background of a photograph, a mobile terminal and a storage medium.
- the dual-camera-based camera background blur (the background blur is to make the depth of field shallow and focus on the subject) is becoming more and more popular.
- the main steps of background blurring are: using binocular cameras to generate a depth map (the depth map is the main image and sub-image taken by the binocular camera to synthesize an image with distance information, and each pixel in the image represents the distance from the scene to the camera. , Also known as depth, so it is called depth map), based on the depth map to segment the foreground and background; and then classify the background according to the depth value, each level is subjected to different blur and smoothing processing and superimposed with the foreground to achieve the background blur effect.
- a depth map is the main image and sub-image taken by the binocular camera to synthesize an image with distance information, and each pixel in the image represents the distance from the scene to the camera.
- depth map also known as depth, so it is called depth map
- the foreground and background segmentation based on the depth map, and the multi-level blurring and superposition of the background based on the depth map have certain defects.
- the estimated depth map of the dual camera may be inaccurate (the depth map is taken by the main and sub cameras of the binocular camera).
- Image estimated resulting in poor or wrong segmentation of the foreground and background, and the use of depth distance information (ie, depth value) hierarchical multi-level blur will make the adjacent areas appear inconsistent with the blur level, the foreground and the blur
- the superimposed background will appear inconsistent in the transition at the edge or have a halo effect; the above-mentioned defects will eventually cause the visual effect of the blurred picture to be poor.
- the main purpose of the present disclosure is to provide a method for blurring the background of a photograph, a mobile terminal, and a storage medium, which aims to solve the problem of inaccurate segmentation of the foreground and background in the prior art and the inconsistency of the blur levels between adjacent background blocks.
- the edges that overlap with the background have halos or unnatural transitions.
- the present disclosure provides a method for blurring the background of a photograph, and the method includes the following steps:
- the segmented picture is blurred to obtain a picture with a blurred background.
- the acquiring a depth map includes:
- the first picture and the second picture are combined into a depth map.
- the preprocessing and segmentation of the depth map includes:
- the depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
- said performing blurring processing on the segmented picture includes:
- the size of the blur kernel used for blurring is determined according to the foreground depth value parameter and the background depth value parameter, and the background blurring is performed on the first picture according to the blur kernel size.
- the first picture is a picture taken by a first camera group of a mobile terminal, and the first camera group includes one or more cameras;
- the second picture is a picture taken by a second camera group of the mobile terminal, and the second camera group includes one or more cameras;
- At least one of the cameras of the first camera group is different from the cameras of the second camera group.
- the preprocessing includes:
- the depth map after edge-preserving filtering processing is subjected to a median filtering operation.
- the segmentation of the preprocessed depth map to obtain a foreground mask and a background mask specifically includes:
- the foreground mask and the background mask are obtained according to a preset segmentation threshold.
- the obtaining the foreground mask and the background mask according to a preset segmentation threshold specifically includes:
- the pixel points in the depth map whose pixel values are less than or equal to the segmentation threshold are classified as the background mask.
- the determining a foreground depth value parameter and a background depth value parameter respectively according to the foreground mask and the background mask specifically includes:
- the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
- the determining a foreground depth value parameter and a background depth value parameter respectively according to the foreground mask and the background mask specifically includes:
- the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
- the method for blurring a photographed background wherein the determining the size of the blur kernel used for blurring according to the foreground depth value parameter and the background depth value parameter specifically includes:
- the blur kernel of the foreground is a Gaussian kernel
- the blur radius Fg_r of the current pixel is:
- Fg_level is the blur level
- depthVal is the depth value
- Fg_mean is the mean value of the foreground depth value
- Fg_std is the mean square error of the foreground depth value
- the size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
- dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;
- the blur kernel of the background is a defocus blur kernel
- the blur radius Bg_r of the current pixel is:
- Bg_r Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);
- Bg_level is the number of blur levels
- Bg_max is the maximum background depth value
- Bg_min is the minimum background depth value
- the size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
- a and b represent weight values, and 0 ⁇ a ⁇ 1,b>1.
- the Fg_level is 2
- the background blurring is a pixel-by-pixel background blurring.
- the method for blurring the background of photographing wherein the method for blurring the background of photographing in the present disclosure further includes:
- the foreground picture and the blurred background picture are superimposed and merged in the transition zone to obtain a picture with a blurred background.
- the preprocessing of the depth map to obtain the foreground image and the background image includes:
- the foreground picture and the background picture are obtained.
- the method for blurring the background of a photograph, wherein the blurring the background picture to obtain a blurring background picture includes:
- the step-by-point multi-level blurring of the first background picture to obtain the second background picture includes:
- the blur radius of the pixel in the first background picture is calculated.
- the method for blurring a photographed background wherein, after calculating a blur radius of a pixel in the first background picture according to the linear distance and the value of the blur degree, the method further includes:
- blur processing is performed on the pixel point.
- the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
- the method further includes:
- the present disclosure also provides a mobile terminal, wherein the mobile terminal includes: a memory, a processor, and a camera background blur program stored in the memory and running on the processor When the photographing background blurring program is executed by the processor, the steps of the aforementioned photographing background blurring method are implemented.
- the present disclosure also provides a storage medium, wherein the storage medium stores a photographing background blurring program, and the photographing background blurring program is executed by a processor to realize the aforementioned photographing background blurring.
- the steps of the chemical method are also provided.
- the present disclosure performs preprocessing and segmentation processing on the depth map by acquiring the depth map; performing blurring processing on the segmented picture to obtain a picture with a blurry background.
- the present disclosure processes the taken pictures, so that the taken pictures have the effect of multi-level blurring of the background, and the effect of the background blurring can be improved.
- FIG. 1 is a flowchart of a preferred embodiment of a method for blurring a photographed background of the present disclosure
- FIG. 2 is a flowchart of the first blurring method in the preferred embodiment of the photo background blurring method of the present disclosure
- FIG. 3 is a depth map after preprocessing in the first blurring method in the preferred embodiment of the photographing background blurring method of the present disclosure
- FIG. 5 is a flowchart of the second blurring mode in the preferred embodiment of the photo background blurring method of the present disclosure
- FIG. 6 is a schematic diagram of the operating environment of the preferred embodiment of the mobile terminal of the present disclosure.
- the method for blurring the background of a photograph includes the following steps:
- Step S100 Obtain a depth map, and perform preprocessing and segmentation processing on the depth map;
- Step S200 Perform blurring processing on the divided picture to obtain a picture with a blurred background.
- the photographing background blurring method includes the following steps:
- Step S10 Synthesize the first picture and the second picture into a depth map.
- the first picture is a picture taken by a first camera group of a mobile terminal.
- the first camera group includes one or more cameras;
- the second picture is a picture taken by a mobile terminal.
- the photo background blurring method of the present disclosure is applied to a mobile terminal (such as the most commonly used smart phone, but also other smart devices with dual cameras to take photos), the mobile terminal includes a main camera (that is, the first camera group) and The most obvious effect of the secondary camera (that is, the second camera group), dual camera or binocular camera is the excellent background blur effect.
- the first picture and the second picture taken at different angles can also be obtained by two devices at the same time stamp, for example, two The first picture and the second picture of different angles obtained by a mobile phone at the same time, the present disclosure is preferably the first picture and the second picture of different angles obtained by the main camera and the auxiliary camera on a mobile terminal at the same time), for example, taken by the main camera Is the first picture, the second picture is taken by the secondary camera, and the first picture and the second picture are synthesized into a depth map.
- the depth map is the main picture and the auxiliary picture taken by a binocular camera with a distance
- the image of the information (each pixel value represents the distance from the object to the xy plane of the camera), and the depth image is also called the range image, which refers to the distance from the image collector to the scene
- the distance (depth) of the point is used as the image of the pixel value, which directly reflects the geometric shape of the visible surface of the scene; and the depth map is estimated from the pictures taken by the primary and secondary cameras of the binocular camera.
- step S20 the depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
- the depth map After synthesizing the depth map, the depth map needs to be preprocessed.
- the purpose of the preprocessing is to obtain a uniform depth map, that is, to preprocess the depth map to make the depth map more uniform and consistent (uniform
- the consistent feature is that the depth value in the depth map changes smoothly, and the local area has a smaller change in depth value, as shown in Figure 3.
- the preprocessing specifically includes: performing edge-preserving filtering processing on the depth map, and the edge-preserving filtering processing is used to protect the edges of the image (that is, the depth map), and the edge-preserving filtering is a filtering method.
- the edge information in the image can be effectively preserved in the process.
- the edge-preserving filter can be processed by the edge-preserving filter; the depth map after the edge-preserving filter processing is subjected to the median filtering operation, and the median filtering operation is used
- median filtering is a non-linear smoothing technique, which sets the gray value of each pixel to the median of the gray values of all pixels in a certain neighborhood window at that point.
- Value filtering is a nonlinear signal processing technology that can effectively suppress noise based on the sorting statistical theory.
- the basic principle of median filtering is to use the value of a point in a digital image or digital sequence with the value of each point in a neighborhood of that point. The median value is replaced, so that the surrounding pixel values are close to the true value, thereby eliminating isolated noise points.
- the depth map after the median filtering operation is segmented, using OTSU (Otsu Method or Maximum Between-Class Variance Method, an efficient algorithm for binarizing the image, using a threshold to divide the image into foreground and Background) Roughly segment the depth map, obtain a preset segmentation threshold (T), and obtain the foreground mask and the background mask according to the preset segmentation threshold.
- OTSU Otsu Method or Maximum Between-Class Variance Method, an efficient algorithm for binarizing the image, using a threshold to divide the image into foreground and Background
- T preset segmentation threshold
- pixels in the depth map with pixel values greater than the segmentation threshold are classified as the foreground mask; pixels in the depth map with pixel values less than or equal to the segmentation threshold are classified as The background mask.
- the foreground mask represents a logo image, in which pixels only have 0 and 1, a pixel of 1 indicates that the point belongs to the foreground, and a pixel of 0 indicates that the point belongs to the background;
- the background mask (Bg_mask) ) Is the inverse of the foreground mask (Fg_mask).
- Step S30 Determine a foreground depth value parameter and a background depth value parameter according to the foreground mask and the background mask, respectively.
- the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are counted respectively; that is, the foreground depth value parameter includes the mean value of the foreground depth value (Fg_mean) and the mean square error of the foreground depth value (Fg_std).
- the background depth value parameter includes a maximum background depth value (Bg_max) and a minimum background depth value (Bg_min).
- the corresponding depth map pixel can be obtained according to the position of the pixel with the flag of 1 in the foreground mask (Fg_mask), the pixel values of all the foreground pixels in the depth map are counted, the average value is calculated, and the variance is calculated, and the background mask After obtaining the pixel values of all background pixels in the depth map, the mask (Bg_mask) can obtain the maximum background depth value and the minimum background depth value.
- Step S40 Determine a blur kernel size for blurring according to the foreground depth value parameter and the background depth value parameter, and perform background blurring on the first picture according to the blur kernel size.
- the background blur is a pixel-by-pixel background blur.
- determining the size of the blur kernel used for blurring according to the foreground depth value parameter and the background depth value parameter specifically includes:
- the blur kernel of the foreground is a Gaussian kernel (the Gaussian kernel means that the weight value in the kernel obeys a two-dimensional discrete Gaussian distribution), then the blur radius Fg_r of the current pixel is:
- Fg_level is the blur level
- depthVal is the depth value
- Fg_mean is the mean value of the foreground depth value
- Fg_std is the mean square error of the foreground depth value
- Fg_level is 2;
- the size of the fuzzy kernel (a fuzzy kernel is a square with a weight value) is (2r+1)*(2r+1), where each weight (each weight represents a different weight value in a kernel, the weight is a value, the reaction
- the calculation formula of the importance of a certain pixel is as follows:
- dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;
- the blur radius Bg_r of the current pixel is:
- Bg_r Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);
- Bg_level is the number of blur levels
- Bg_max is the maximum background depth value
- Bg_min is the minimum background depth value
- Bg_level is 11;
- the size of the fuzzy kernel (a fuzzy kernel is a square with a weight value) is (2r+1)*(2r+1), and the calculation formula for each weight is as follows:
- the first picture taken by the main camera of the mobile terminal (the picture of the secondary camera does not need to be processed, and the function of the secondary camera is to cooperate with the main camera to generate a depth map), and perform pixel-by-pixel blurring according to the corresponding depth map traversal ,
- the blur kernel of each pixel depends on the corresponding depth value depthVal, and the blur kernel is selected as described above.
- the blur operation between pixels is independent, and multi-threaded parallel acceleration calculation can be used to obtain the background blur effect map, as shown in FIG. 4.
- the present disclosure can improve the visual effect of the blurred background of the dual-camera photography of the mobile terminal, and reduce the memory consumption of the mobile terminal.
- the present disclosure directly performs blur processing based on different blur kernel sizes on the original images (such as the first picture and the second picture), the depth of the foreground area is small, and the blur kernel size is basically 1, which is essentially equivalent to no blurring; Even if there is an abnormal depth value, a small blur core (3x3) will not affect the final visual effect; do not do the layered multi-level blur processing superimposition of the background, avoiding the halo or the edge of the foreground and the background superimposed Transition unnatural phenomenon, real-time multi-level background blur effect.
- the layered multi-level background blur using depth map and the superposition of foreground and background have certain defects, which affect the effect of photo shooting. Can not meet the real-time nature of mobile phone photography.
- the effect of background blur is mainly improved in three aspects: 1. Through the stretching operation and reverse stretching operation of the pixels of the background picture, the blurred background has a certain degree of Spot effect; 2. Taking the center of the foreground contour as the center of the circle, the circle is diffused outward to increase the radius of the blur core to blur point by point, achieving the purpose of multi-level blur but not layering, and solves the multi-level blur Improve the unnatural problem of edge transition of background superimposition; 3.
- the method of etching and Gaussian blur smoothing of the foreground mask generates a transition zone between the foreground and the background, and merges the foreground and background of the transition zone, effectively alleviating the abrupt edge transition and the appearance of black Issues such as edges and halos.
- the embodiment of the present disclosure provides a second blurring method in the background blurring method for photographing. As shown in FIG. 5, the method includes:
- the depth map is preprocessed, the foreground image F_Img and the background image B_Img are segmented, and the corresponding foreground mask F_mask and background mask B_mask are obtained, and the foreground image and background image are obtained according to the foreground mask F_mask and the background mask B_mask.
- the depth map may be a picture obtained by taking a picture and processing it.
- the foreground picture can be the subject in the captured picture, the object close to the lens, the object that the user wants to take, or the focus of the mobile phone camera, etc.
- the background picture is the picture excluding the foreground area in the picture. For example, if a person is photographed, the area of the person in the picture is the foreground picture, and the rest of the area is the background picture.
- step S1 includes:
- median filtering is performed on the first depth picture in step S12.
- level set segmentation By performing level set segmentation on the second depth image, the contour of the foreground area is obtained, and the foreground mask and the background mask are obtained according to the contour of the foreground area.
- the first depth picture and the second depth picture are both depth pictures during processing.
- the blurred background picture is a blurred background picture.
- step S2 the pixelValue in the background image that exceeds the first threshold is stretched and transformed, and then the stretched background image is subjected to point-by-point multi-level blurring, and finally the blurring background image is reverse-stretched Transform to get a blurred background image.
- step S2 includes:
- the first background picture is a background picture after stretching and transformation. Perform stretching transformation on the pixel values in the background image that exceed the first threshold, where the representation form of the stretching transformation is as follows:
- the second background picture is a background picture that is blurred by point-by-point multi-level blur.
- the centroid point of the foreground contour is calculated, and the linear distance between each pixel point and the centroid point in the stretched background image is calculated.
- a point-by-point multi-level blurring method is used to solve the problem that the hierarchical multi-level blurring using depth distance information in the prior art will cause inconsistent blur levels in adjacent areas, and more blur levels will cause The processing time is too long to meet the real-time performance of the mobile phone, which makes the background blurry and the transition more natural.
- step S22 includes:
- centroid point (x0, y0) of the foreground contour can be calculated according to the contour of the foreground in the foreground picture.
- S222 Calculate the linear distance between each pixel point in the first background picture and the centroid point according to the coordinates of the pixel points in the first background picture and the coordinates of the centroid point.
- the first background picture is a stretched background picture, and it is calculated according to the coordinates (i, j) of the pixel points in the first background picture and the coordinates (x0, y0) of the centroid point The linear distance between each pixel point (i, j) and the centroid point (x0, y0) in the first background picture.
- S223 Calculate the blur radius of the pixel in the first background picture according to the straight line distance and the blur degree value.
- centroid point (x0, y0) of the foreground contour is calculated according to the contour of the foreground, and the linear distance dist between all (i, j) and (x0, y0) is calculated.
- the characterization form of the radius of the pixel point blur is:
- disti, j is the distance between the pixel point of the background image coordinate (i, j) and (x0, y0), maxDist is the maximum value of all straight-line distances, baseRadius is the value of the degree of blur, baseRadius is taken according to the degree of blur Value, for example: baseRadius can be 15.
- step S223 the method further includes:
- S224 Generate a blur kernel of the pixel point according to the blur radius of the pixel point.
- S225 Perform blur processing on the pixel according to the blur kernel of the pixel.
- the pixels in the first background picture are traversed, the corresponding blur kernel is generated according to the blur radius of the pixel, and the pixel is blurred.
- the pixel-by-pixel traversal is relatively time-consuming, and the blur operation between pixels is independent, and multi-threaded parallel processing can be used to accelerate processing.
- the second background picture is a background picture that is blurred by point-by-point multi-level blur.
- the characterization form of the inverse stretch transformation is the inverse function of the above-mentioned stretch transformation expression.
- the mask F_mask of the foreground area is processed by etching and Gaussian blur, and a transition zone with pixel values ranging from 0 to 255 (not including 0 and 255) is generated at the edge of the mask and normalized To 0 ⁇ 1, the transition zone is the edge area adjacent to the foreground and the background.
- the foreground pixel value of the binary mask image is 255, and the background pixel value is 0.
- a smoothing is performed on the edge part adjacent to the foreground and background.
- Processing such as Gaussian blur processing, makes the pixel values of the foreground and background edge parts between 0 and 255.
- the area where the pixel value is between 0 and 255 is called the transition zone.
- step S4 includes:
- the pixels belonging to the foreground picture are replaced with the pixels of the corresponding position in the foreground picture, and the pixels belonging to the background picture in the non-transition zone area of the taken picture are replaced with the background
- the pixels at the corresponding positions in the picture are replaced.
- the background of the picture in the non-transition zone is the blurred background.
- the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
- the pixel value of the F_mask processed in step S3 of the pixel of the transition zone area is w
- the pixel value of the transition zone area is F_pixel in the foreground image
- the pixel point of the transition zone area is The pixel value of the blurred background image
- the processed foreground and background are merged by generating a transition zone between the foreground and the background, so as to alleviate the abrupt edge transition and avoid the problem of black borders and halos appearing on the boundary between the foreground and the background.
- the present disclosure also provides a mobile terminal correspondingly, and the mobile terminal includes a processor 10, a memory 20 and a display 30.
- FIG. 6 only shows part of the components of the mobile terminal, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.
- the mobile terminal also includes a first camera group and a second camera group; the first camera group and the second camera group are used to acquire two first pictures and second pictures taken from different angles; the first The picture is a picture taken by the first camera group, the first camera group includes one or more cameras; the second picture is a picture taken by the second camera group, the second camera group includes One or more cameras; at least one of the cameras of the first camera group is different from the cameras of the second camera group.
- the memory 20 may be an internal storage unit of the mobile terminal in some embodiments, such as a hard disk or a memory of the mobile terminal. In other embodiments, the memory 20 may also be an external storage device of the mobile terminal, such as a plug-in hard disk equipped on the mobile terminal, a smart media card (SMC), and a secure digital (Secure Digital). Digital, SD) card, flash card, etc. Further, the memory 20 may also include both an internal storage unit of the mobile terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the mobile terminal, such as the program code of the installed mobile terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a photographing background blurring program 40, and the photographing background blurring program 40 can be executed by the processor 10, so as to realize the photographing background blurring method in the present disclosure.
- the processor 10 may be a central processing unit (CPU), microprocessor or other data processing chip in some embodiments, and is used to run the program code or process data stored in the memory 20, for example Perform the method of blurring the background of the photograph and so on.
- CPU central processing unit
- microprocessor or other data processing chip in some embodiments, and is used to run the program code or process data stored in the memory 20, for example Perform the method of blurring the background of the photograph and so on.
- the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc.
- the display 30 is used for displaying information on the mobile terminal and for displaying a visualized user interface.
- the components 10-30 of the mobile terminal communicate with each other via a system bus.
- the segmented picture is blurred to obtain a picture with a blurred background.
- the acquiring depth map includes:
- the first picture and the second picture are combined into a depth map.
- the preprocessing and segmentation processing of the depth map includes:
- the depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
- the blurring processing of the divided picture includes:
- the size of the blur kernel used for blurring is determined according to the foreground depth value parameter and the background depth value parameter, and the background blurring is performed on the first picture according to the blur kernel size.
- the first picture is a picture taken by a first camera group of the mobile terminal, and the first camera group includes one or more cameras;
- the second picture is a picture taken by a second camera group of the mobile terminal, and the second camera group includes one or more cameras;
- At least one of the cameras of the first camera group is different from the cameras of the second camera group.
- the preprocessing includes:
- the depth map after edge-preserving filtering processing is subjected to a median filtering operation.
- the foreground mask and the background mask are obtained according to a preset segmentation threshold.
- the pixel points in the depth map whose pixel values are less than or equal to the segmentation threshold are classified as the background mask.
- the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
- the blur kernel of the foreground is a Gaussian kernel
- the blur radius Fg_r of the current pixel is:
- Fg_level is the blur level
- depthVal is the depth value
- Fg_mean is the mean value of the foreground depth value
- Fg_std is the mean square error of the foreground depth value
- the size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
- dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;
- the blur kernel of the background is a defocus blur kernel
- the blur radius Bg_r of the current pixel is:
- Bg_r Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);
- Bg_level is the number of blur levels
- Bg_max is the maximum background depth value
- Bg_min is the minimum background depth value
- the size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
- a and b represent weight values, and 0 ⁇ a ⁇ 1,b>1.
- the background blur is a pixel-by-pixel background blur.
- the foreground picture and the blurred background picture are superimposed and merged in the transition zone to obtain a picture with a blurred background.
- the foreground picture and the background picture are obtained.
- the blur radius of the pixel in the first background picture is calculated.
- the method After calculating the blur radius of the pixel in the first background picture according to the straight line distance and the blur degree value, the method includes:
- blur processing is performed on the pixel point.
- the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
- the present disclosure also provides a storage medium, wherein the storage medium stores a photographing background blurring program, and the photographing background blurring program is executed by a processor to realize the steps of the photographing background blurring method as described above.
- the present disclosure provides a method for blurring a photographed background, a mobile terminal, and a storage medium.
- the method includes: acquiring a depth map, performing preprocessing and segmentation processing on the depth map; and performing a virtual image on the segmented picture. After the bokeh process, the picture with the blurred background is obtained.
- the present disclosure processes the taken pictures, so that the taken pictures have the effect of multi-level blurring of the background, and the effect of the background blurring can be improved.
- the processes in the methods of the above-mentioned embodiments can be implemented by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer program.
- the program may include the processes of the foregoing method embodiments when executed.
- the storage medium mentioned may be a memory, a magnetic disk, an optical disk, and the like.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
A photographing background blurring method, a mobile terminal, and a storage medium. The method comprises: obtaining a depth image, and preprocessing and segmenting the depth image; blurring a segmented image to obtain an image subjected to background blurring. According to the method, a captured image is processed, so that the captured image has the effect of multi-level background blurring, and the effect of background blurring is improved.
Description
优先权priority
本公开要求于申请日为2019年12月30日提交中国专利局、申请号为“201911397607.7”、发明名称为“一种图片的背景虚化方法及装置、计算机设备、存储介质”和申请日为2020年02月10日提交中国专利局、申请号为“202010085333.4”、发明名称为“一种拍照背景虚化方法、移动终端及存储介质”的中国专利发明的优先权,其全部内容通过引用结合在本公开中。This disclosure requires that it be submitted to the Chinese Patent Office on December 30, 2019, the application number is "201911397607.7", the title of the invention is "a method and device for background blurring of pictures, computer equipment, and storage media" and the application date is The priority of a Chinese patent invention filed with the Chinese Patent Office on February 10, 2020, with the application number "202010085333.4" and the invention title "A method for blurring the background of a photograph, a mobile terminal and a storage medium", the entire content of which is incorporated by reference In this disclosure.
本公开涉及图像处理技术领域,尤其涉及一种拍照背景虚化方法、移动终端及存储介质。The present disclosure relates to the technical field of image processing, and in particular to a method for blurring the background of a photograph, a mobile terminal and a storage medium.
现在的智能手机的拍照功能中,基于双摄像头的拍照背景虚化(背景虚化就是使景深变浅,使焦点聚集在主题上)功能越来越流行。Among the current smartphone camera functions, the dual-camera-based camera background blur (the background blur is to make the depth of field shallow and focus on the subject) is becoming more and more popular.
目前,背景虚化主要步骤为:利用双目摄像头生成深度图(深度图是采用双目摄像头拍摄的主图和副图合成具有距离信息的图像,图像中的每一个像素表示场景到摄像头的距离,也称为深度,所以叫深度图),基于深度图分割出前景和背景;再根据深度值对背景做分级,每一级做不同的模糊平滑处理后与前景叠加实现背景虚化效果。At present, the main steps of background blurring are: using binocular cameras to generate a depth map (the depth map is the main image and sub-image taken by the binocular camera to synthesize an image with distance information, and each pixel in the image represents the distance from the scene to the camera. , Also known as depth, so it is called depth map), based on the depth map to segment the foreground and background; and then classify the background according to the depth value, each level is subjected to different blur and smoothing processing and superimposed with the foreground to achieve the background blur effect.
但是,基于深度图的前景与背景分割、背景分层多级虚化后叠加等方式都存在一定的缺陷,双摄像头估计的深度图可能不准(深度图是双目摄像头的主副摄像头拍摄的图片估计出来的),导致前景和背景的分割效果不佳或出错,并且利用深度距离信息(即深度值)的分层多级虚化会使得相邻的区域出现虚化等级不一致,前景与虚化后的背景叠加会在边缘处过渡显得不一致、或出现光晕效果;上述缺陷最终都会造成虚化后图片的视觉效果不佳。However, the foreground and background segmentation based on the depth map, and the multi-level blurring and superposition of the background based on the depth map have certain defects. The estimated depth map of the dual camera may be inaccurate (the depth map is taken by the main and sub cameras of the binocular camera). Image estimated), resulting in poor or wrong segmentation of the foreground and background, and the use of depth distance information (ie, depth value) hierarchical multi-level blur will make the adjacent areas appear inconsistent with the blur level, the foreground and the blur The superimposed background will appear inconsistent in the transition at the edge or have a halo effect; the above-mentioned defects will eventually cause the visual effect of the blurred picture to be poor.
因此,现有技术还有待于改进和发展。Therefore, the existing technology needs to be improved and developed.
公开内容Public content
本公开的主要目的在于提供一种拍照背景虚化方法、移动终端及存储介质,旨在解决现有技术中前景和背景分割不准,相邻的背景块之间的虚化等级不一致导致的前景与背景叠加的边缘出现光晕或过渡不自然的问题。The main purpose of the present disclosure is to provide a method for blurring the background of a photograph, a mobile terminal, and a storage medium, which aims to solve the problem of inaccurate segmentation of the foreground and background in the prior art and the inconsistency of the blur levels between adjacent background blocks. The edges that overlap with the background have halos or unnatural transitions.
为实现上述目的,本公开提供一种拍照背景虚化方法,所述拍照背景虚化方法包括如下步骤:In order to achieve the above objective, the present disclosure provides a method for blurring the background of a photograph, and the method includes the following steps:
获取深度图,将所述深度图进行预处理和分割处理;Acquire a depth map, and perform preprocessing and segmentation processing on the depth map;
将分割后的图片进行虚化处理,得到背景虚化后的图片。The segmented picture is blurred to obtain a picture with a blurred background.
所述的拍照背景虚化方法,其中,所述获取深度图包括:In the method for blurring the background of a photograph, the acquiring a depth map includes:
获取第一图片和第二图片;Obtain the first picture and the second picture;
将第一图片和第二图片合成深度图。The first picture and the second picture are combined into a depth map.
所述的拍照背景虚化方法,其中,所述将所述深度图进行预处理和分割处理包括:In the method for blurring a photographed background, the preprocessing and segmentation of the depth map includes:
将所述深度图进行预处理,将预处理后的所述深度图进行分割得到前景掩模和背景掩模。The depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
所述的拍照背景虚化方法,其中,所述将分割后的图片进行虚化处理包括:In the method for blurring the background of a photograph, said performing blurring processing on the segmented picture includes:
根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数;Respectively determining a foreground depth value parameter and a background depth value parameter according to the foreground mask and the background mask;
根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,并根据所述模糊核大小对所述第一图片进行背景虚化。The size of the blur kernel used for blurring is determined according to the foreground depth value parameter and the background depth value parameter, and the background blurring is performed on the first picture according to the blur kernel size.
所述的拍照背景虚化方法,其中,所述第一图片是由移动终端的第一摄像头组拍摄得到的图片,所述第一摄像头组包括一个或者多个摄像头;In the method for blurring a photographed background, the first picture is a picture taken by a first camera group of a mobile terminal, and the first camera group includes one or more cameras;
所述第二图片是由移动终端的第二摄像头组拍摄得到的图片,所述第二摄像头组包括一个或者多个摄像头;The second picture is a picture taken by a second camera group of the mobile terminal, and the second camera group includes one or more cameras;
所述第一摄像头组的摄像头与所述第二摄像头组的摄像头之间至少有一个不相同。At least one of the cameras of the first camera group is different from the cameras of the second camera group.
所述的拍照背景虚化方法,其中,所述预处理包括:In the method for blurring a photographed background, the preprocessing includes:
将所述深度图进行保边滤波处理;Performing edge-preserving filtering processing on the depth map;
将进行保边滤波处理后的所述深度图进行中值滤波操作。The depth map after edge-preserving filtering processing is subjected to a median filtering operation.
所述的拍照背景虚化方法,其中,所述将预处理后的所述深度图进行分割得 到前景掩模和背景掩模,具体包括:In the method for blurring the background of a photograph, the segmentation of the preprocessed depth map to obtain a foreground mask and a background mask specifically includes:
将进行中值滤波操作后的所述深度图进行分割处理;Performing segmentation processing on the depth map after the median filtering operation;
根据预先设置的分割阈值得到所述前景掩模和所述背景掩模。The foreground mask and the background mask are obtained according to a preset segmentation threshold.
所述的拍照背景虚化方法,其中,所述根据预先设置的分割阈值得到所述前景掩模和所述背景掩模,具体包括:In the method for blurring a photographed background, wherein the obtaining the foreground mask and the background mask according to a preset segmentation threshold specifically includes:
将所述深度图中像素值大于所述分割阈值的像素点归为所述前景掩模;Classify pixels with pixel values greater than the segmentation threshold in the depth map as the foreground mask;
将所述深度图中像素值小于或等于所述分割阈值的像素点归为所述背景掩模。The pixel points in the depth map whose pixel values are less than or equal to the segmentation threshold are classified as the background mask.
所述的拍照背景虚化方法,其中,所述根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数,具体包括:In the method for blurring a photographed background, wherein the determining a foreground depth value parameter and a background depth value parameter respectively according to the foreground mask and the background mask specifically includes:
根据所述前景掩模和所述背景掩模分别统计前景深度值均值和前景深度值均方差,以及背景深度值最大值和背景深度值最小值。According to the foreground mask and the background mask, the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
所述的拍照背景虚化方法,其中,所述根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数,具体包括:In the method for blurring a photographed background, wherein the determining a foreground depth value parameter and a background depth value parameter respectively according to the foreground mask and the background mask specifically includes:
根据所述前景掩模和所述背景掩模分别统计前景深度值均值和前景深度值均方差,以及背景深度值最大值和背景深度值最小值。According to the foreground mask and the background mask, the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
所述的拍照背景虚化方法,其中,所述根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,具体包括:The method for blurring a photographed background, wherein the determining the size of the blur kernel used for blurring according to the foreground depth value parameter and the background depth value parameter specifically includes:
确定前景的虚化核为高斯核,则当前像素的虚化半径Fg_r为:It is determined that the blur kernel of the foreground is a Gaussian kernel, then the blur radius Fg_r of the current pixel is:
其中,Fg_level为虚化级数,depthVal为深度值,Fg_mean为前景深度值均值,Fg_std为前景深度值均方差;Among them, Fg_level is the blur level, depthVal is the depth value, Fg_mean is the mean value of the foreground depth value, and Fg_std is the mean square error of the foreground depth value;
模糊核的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
Fg_k=exp(-Fg_r*dist);Fg_k=exp(-Fg_r*dist);
其中,dist为核中点(r,r)与其邻域的欧氏距离;Among them, dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;
确定背景的虚化核为散焦模糊核,则当前像素的虚化半径Bg_r为:It is determined that the blur kernel of the background is a defocus blur kernel, then the blur radius Bg_r of the current pixel is:
Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1) ;Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);
其中,Bg_level为虚化级数,Bg_max为背景深度值最大值,Bg_min为背景深度值最小值;Among them, Bg_level is the number of blur levels, Bg_max is the maximum background depth value, and Bg_min is the minimum background depth value;
模糊核的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
Bg_k=a*dist+b;Bg_k=a*dist+b;
其中,a和b表示权重值,且0<a<1,b>1。Among them, a and b represent weight values, and 0<a<1,b>1.
所述的拍照背景虚化方法,其中,所述Fg_level为2,所述Bg_level为11,a=0.1,b=10。In the method for blurring the background of a photograph, the Fg_level is 2, the Bg_level is 11, a=0.1, and b=10.
所述的拍照背景虚化方法,其中,所述背景虚化为逐像素的背景虚化。In the method for blurring a photographed background, the background blurring is a pixel-by-pixel background blurring.
所述的拍照背景虚化方法,其中,本公开所述拍照背景虚化方法还包括:The method for blurring the background of photographing, wherein the method for blurring the background of photographing in the present disclosure further includes:
对深度图做预处理,得到前景图片和背景图片;Preprocess the depth map to get the foreground image and background image;
对所述背景图片进行虚化,得到虚化背景图片;Blur the background picture to obtain a blurred background picture;
对所述前景图片进行模糊,得到所述前景图片和所述背景图片之间的过渡带;Blur the foreground picture to obtain a transition zone between the foreground picture and the background picture;
将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片。The foreground picture and the blurred background picture are superimposed and merged in the transition zone to obtain a picture with a blurred background.
所述的拍照背景虚化方法,其中,所述对深度图做预处理,得到前景图片和背景图片,包括:In the method for blurring the background of a photograph, the preprocessing of the depth map to obtain the foreground image and the background image includes:
对所述深度图进行保边滤波,得到第一深度图片;Performing edge-preserving filtering on the depth image to obtain a first depth image;
对所述第一深度图片进行中值滤波,得到第二深度图片;Performing median filtering on the first depth picture to obtain a second depth picture;
对所述第二深度图片进行水平集分割,得到前景掩模和背景掩模;Performing level set segmentation on the second depth picture to obtain a foreground mask and a background mask;
根据所述前景掩模和所述背景掩模,得到所述前景图片和所述背景图片。According to the foreground mask and the background mask, the foreground picture and the background picture are obtained.
所述的拍照背景虚化方法,其中,所述对所述背景图片进行虚化,得到虚化背景图片,包括:The method for blurring the background of a photograph, wherein the blurring the background picture to obtain a blurring background picture includes:
对所述背景图片进行拉伸变换,得到第一背景图片;Stretch and transform the background picture to obtain a first background picture;
对所述第一背景图片进行逐点多级虚化,得到第二背景图片;Performing point-by-point multi-level blurring on the first background picture to obtain a second background picture;
对所述第二背景图片做反拉伸变换,得到所述虚化背景图片。Performing an inverse stretch transformation on the second background picture to obtain the blurred background picture.
所述的拍照背景虚化方法,其中,所述对所述第一背景图片进行逐点多级虚化,得到第二背景图片,包括:In the method for blurring the background of a photograph, the step-by-point multi-level blurring of the first background picture to obtain the second background picture includes:
根据所述前景图片,计算所述前景图片的质心点的坐标;Calculating the coordinates of the centroid point of the foreground picture according to the foreground picture;
根据所述第一背景图片中像素点的坐标和所述质心点的坐标,计算所述第一背景图片中每一个像素点与质心点之间的直线距离;Calculating the linear distance between each pixel point in the first background picture and the centroid point according to the coordinates of the pixel points in the first background picture and the coordinates of the centroid point;
根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径。According to the straight line distance and the blur degree value, the blur radius of the pixel in the first background picture is calculated.
所述的拍照背景虚化方法,其中,根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径之后,还包括:The method for blurring a photographed background, wherein, after calculating a blur radius of a pixel in the first background picture according to the linear distance and the value of the blur degree, the method further includes:
根据所述像素点的虚化半径生成所述像素点的模糊核;Generating a blur kernel of the pixel according to the blur radius of the pixel;
根据所述像素点的模糊核,对所述像素点做模糊处理。According to the blur kernel of the pixel point, blur processing is performed on the pixel point.
所述的拍照背景虚化方法,其中,所述将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片,包括:In the method for blurring a photographed background, wherein the superimposing the foreground picture and the blurring background picture and fusing them in the transition zone to obtain a picture with a blurring background includes:
在非过渡带区域,用所述前景图片或者所述虚化背景图片中的像素点对拍摄图片中对应的像素点进行替换;In the non-transition zone area, replace the corresponding pixels in the captured picture with the pixels in the foreground picture or the blurred background picture;
在过渡带区域,根据模糊值、前景图片的像素值和虚化背景图片的像素值,得到过渡带区域融合后的像素值,其中,所述模糊值为对所述前景图片腐蚀和模糊后的像素值。In the transition zone area, according to the blur value, the pixel value of the foreground picture, and the pixel value of the blurred background picture, the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
所述的拍照背景虚化方法,其中,所述对深度图做预处理,得到前景图片和背景图片之前,还包括:In the method for blurring the background of a photograph, before the preprocessing of the depth map to obtain the foreground image and the background image, the method further includes:
获取拍摄图片和所述深度图。Obtain the photographed picture and the depth map.
此外,为实现上述目的,本公开还提供一种移动终端,其中,所述移动终端包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的拍照背景虚化程序,所述拍照背景虚化程序被所述处理器执行时实现如上所述的拍照背景虚化方法的步骤。In addition, in order to achieve the above object, the present disclosure also provides a mobile terminal, wherein the mobile terminal includes: a memory, a processor, and a camera background blur program stored in the memory and running on the processor When the photographing background blurring program is executed by the processor, the steps of the aforementioned photographing background blurring method are implemented.
此外,为实现上述目的,本公开还提供一种存储介质,其中,所述存储介质存储有拍照背景虚化程序,所述拍照背景虚化程序被处理器执行时实现如上所述的拍照背景虚化方法的步骤。In addition, in order to achieve the above-mentioned object, the present disclosure also provides a storage medium, wherein the storage medium stores a photographing background blurring program, and the photographing background blurring program is executed by a processor to realize the aforementioned photographing background blurring. The steps of the chemical method.
本公开通过获取深度图,将所述深度图进行预处理和分割处理;将分割后的图片进行虚化处理,得到背景虚化后的图片。本公开对拍摄的图片进行处理,可以使拍摄的照片具有背景多级虚化的效果,提升背景虚化的效果。The present disclosure performs preprocessing and segmentation processing on the depth map by acquiring the depth map; performing blurring processing on the segmented picture to obtain a picture with a blurry background. The present disclosure processes the taken pictures, so that the taken pictures have the effect of multi-level blurring of the background, and the effect of the background blurring can be improved.
图1是本公开拍照背景虚化方法的较佳实施例的流程图;FIG. 1 is a flowchart of a preferred embodiment of a method for blurring a photographed background of the present disclosure;
图2是本公开拍照背景虚化方法的较佳实施例中第一种虚化方式的流程图;2 is a flowchart of the first blurring method in the preferred embodiment of the photo background blurring method of the present disclosure;
图3是本公开拍照背景虚化方法的较佳实施例中第一种虚化方式中进行预处理后的深度图;FIG. 3 is a depth map after preprocessing in the first blurring method in the preferred embodiment of the photographing background blurring method of the present disclosure;
图4是本公开拍照背景虚化方法的较佳实施例中第一种虚化方式中进行背景虚化后的效果图;4 is an effect diagram after background blurring in the first blurring method in the preferred embodiment of the photographing background blurring method of the present disclosure;
图5是本公开拍照背景虚化方法的较佳实施例中第二种虚化方式中的流程图;FIG. 5 is a flowchart of the second blurring mode in the preferred embodiment of the photo background blurring method of the present disclosure;
图6为本公开移动终端的较佳实施例的运行环境示意图。FIG. 6 is a schematic diagram of the operating environment of the preferred embodiment of the mobile terminal of the present disclosure.
为使本公开的目的、技术方案及优点更加清楚、明确,以下参照附图并举实施例对本公开进一步详细说明。应当理解,此处所描述的具体实施例仅仅用以解释本公开,并不用于限定本公开。In order to make the objectives, technical solutions, and advantages of the present disclosure clearer and clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure, but not used to limit the present disclosure.
本公开较佳实施例所述的拍照背景虚化方法,如图1所示,所述拍照背景虚化方法包括以下步骤:As shown in FIG. 1, the method for blurring the background of a photograph according to a preferred embodiment of the present disclosure includes the following steps:
步骤S100、获取深度图,将所述深度图进行预处理和分割处理;Step S100: Obtain a depth map, and perform preprocessing and segmentation processing on the depth map;
步骤S200、将分割后的图片进行虚化处理,得到背景虚化后的图片。Step S200: Perform blurring processing on the divided picture to obtain a picture with a blurred background.
本公开较佳实施例所述的拍照背景虚化方法中第一种虚化方式,如图2所示,所述拍照背景虚化方法包括以下步骤:The first blurring method in the photographing background blurring method according to the preferred embodiment of the present disclosure, as shown in FIG. 2, the photographing background blurring method includes the following steps:
步骤S10、将第一图片和第二图片合成深度图。Step S10: Synthesize the first picture and the second picture into a depth map.
先获取第一图片和第二图片,所述第一图片是由移动终端的第一摄像头组拍摄得到的图片,所述第一摄像头组包括一个或者多个摄像头;所述第二图片是由移动终端的第二摄像头组拍摄得到的图片,所述第二摄像头组包括一个或者多个摄像头;所述第一摄像头组的摄像头与所述第二摄像头组的摄像头之间至少有一个不相同。First obtain a first picture and a second picture. The first picture is a picture taken by a first camera group of a mobile terminal. The first camera group includes one or more cameras; the second picture is a picture taken by a mobile terminal. A picture taken by a second camera group of the terminal, where the second camera group includes one or more cameras; at least one of the cameras of the first camera group is different from the cameras of the second camera group.
本公开的拍照背景虚化方法应用于移动终端(例如最常使用的智能手机,还包括其他具有双摄像头拍照的智能设备),所述移动终端包括主摄像头(即所述第一摄像头组)和副摄像头(即所述第二摄像头组),双摄像头或者叫双目摄像头最明显的效果就是有出色的背景虚化效果,首先,通过所述主摄像头和所述副 摄像头拍摄图片,并获取两张不同角度拍摄的第一图片和第二图片(需要说明的是,本公开中获取两张不同角度拍摄的第一图片和第二图片还可以是两个设备在同一时间戳获取的,例如两台手机同时获取的不同角度的第一图片和第二图片,本公开优选为一个移动终端上主摄像头和副摄像头同时获取的不同角度的第一图片和第二图片),例如所述主摄像头拍摄的为第一图片,所述副摄像头拍摄为第二图片,并将所述第一图片和所述第二图片合成深度图,深度图是采用双目摄像头拍摄的主图和副图合成具有距离信息的图像(其每个像素值代表的是物体到相机xy平面的距离),且深度图(depth image)也被称为距离影像(range image),是指将从图像采集器到场景中各点的距离(深度)作为像素值的图像,它直接反映了景物可见表面的几何形状;且所述深度图是双目摄像头的主副摄像头拍摄的图片估计出来的。The photo background blurring method of the present disclosure is applied to a mobile terminal (such as the most commonly used smart phone, but also other smart devices with dual cameras to take photos), the mobile terminal includes a main camera (that is, the first camera group) and The most obvious effect of the secondary camera (that is, the second camera group), dual camera or binocular camera is the excellent background blur effect. First, take pictures through the main camera and the secondary camera, and obtain two The first picture and the second picture taken at different angles (it should be noted that in this disclosure, the first picture and the second picture taken at different angles can also be obtained by two devices at the same time stamp, for example, two The first picture and the second picture of different angles obtained by a mobile phone at the same time, the present disclosure is preferably the first picture and the second picture of different angles obtained by the main camera and the auxiliary camera on a mobile terminal at the same time), for example, taken by the main camera Is the first picture, the second picture is taken by the secondary camera, and the first picture and the second picture are synthesized into a depth map. The depth map is the main picture and the auxiliary picture taken by a binocular camera with a distance The image of the information (each pixel value represents the distance from the object to the xy plane of the camera), and the depth image is also called the range image, which refers to the distance from the image collector to the scene The distance (depth) of the point is used as the image of the pixel value, which directly reflects the geometric shape of the visible surface of the scene; and the depth map is estimated from the pictures taken by the primary and secondary cameras of the binocular camera.
步骤S20、将所述深度图进行预处理,将预处理后的所述深度图进行分割得到前景掩模和背景掩模。In step S20, the depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
在合成所述深度图之后,需要对所述深度图进行预处理,所述预处理的目的是为了得到均匀一致的深度图,即对深度图做预处理使得深度图比较均匀和一致性(均匀一致的特点是深度图中的深度值变化平滑,局部区域是深度值变化较小),如图3所示。After synthesizing the depth map, the depth map needs to be preprocessed. The purpose of the preprocessing is to obtain a uniform depth map, that is, to preprocess the depth map to make the depth map more uniform and consistent (uniform The consistent feature is that the depth value in the depth map changes smoothly, and the local area has a smaller change in depth value, as shown in Figure 3.
其中,所述预处理具体包括:将所述深度图进行保边滤波处理,所述保边滤波处理用于保护图像(即深度图)边缘,其中,保边滤波是一种滤波方法,在滤波过程中能够有效的保留图像中的边缘信息,例如可以通过保边滤波器进行保边滤波处理;将进行保边滤波处理后的所述深度图进行中值滤波操作,所述中值滤波操作用于去除图像毛边,其中,中值滤波是一种非线性平滑技术,它将每一像素点的灰度值设置为该点某邻域窗口内的所有像素点灰度值的中值,同时中值滤波是基于排序统计理论的一种能有效抑制噪声的非线性信号处理技术,中值滤波的基本原理是把数字图像或数字序列中一点的值用该点的一个邻域中各点值的中值代替,让周围的像素值接近的真实值,从而消除孤立的噪声点。Wherein, the preprocessing specifically includes: performing edge-preserving filtering processing on the depth map, and the edge-preserving filtering processing is used to protect the edges of the image (that is, the depth map), and the edge-preserving filtering is a filtering method. The edge information in the image can be effectively preserved in the process. For example, the edge-preserving filter can be processed by the edge-preserving filter; the depth map after the edge-preserving filter processing is subjected to the median filtering operation, and the median filtering operation is used To remove image burrs, median filtering is a non-linear smoothing technique, which sets the gray value of each pixel to the median of the gray values of all pixels in a certain neighborhood window at that point. Value filtering is a nonlinear signal processing technology that can effectively suppress noise based on the sorting statistical theory. The basic principle of median filtering is to use the value of a point in a digital image or digital sequence with the value of each point in a neighborhood of that point. The median value is replaced, so that the surrounding pixel values are close to the true value, thereby eliminating isolated noise points.
进一步地,将进行中值滤波操作后的所述深度图进行分割处理,采用OTSU(大津法或最大类间方差法,一种对图像进行二值化的高效算法,利用阈值将图像分成前景和背景)粗略分割所述深度图,获取预先设置的分割阈值(T),根 据预先设置的分割阈值得到所述前景掩模和所述背景掩模。Further, the depth map after the median filtering operation is segmented, using OTSU (Otsu Method or Maximum Between-Class Variance Method, an efficient algorithm for binarizing the image, using a threshold to divide the image into foreground and Background) Roughly segment the depth map, obtain a preset segmentation threshold (T), and obtain the foreground mask and the background mask according to the preset segmentation threshold.
其中,所述分割阈值(T)是一个像素值,比如:一幅图像的像素在0~255之间,如果T=128,将比T小或者相同的像素点归类为背景,比T大的像素点归类为前景。Wherein, the segmentation threshold (T) is a pixel value, for example: the pixels of an image are between 0 and 255, if T=128, the pixels smaller than T or the same are classified as background, and larger than T The pixels are classified as foreground.
因此,本公开中,将所述深度图中像素值大于所述分割阈值的像素点归为所述前景掩模;将所述深度图中像素值小于或等于所述分割阈值的像素点归为所述背景掩模。Therefore, in the present disclosure, pixels in the depth map with pixel values greater than the segmentation threshold are classified as the foreground mask; pixels in the depth map with pixel values less than or equal to the segmentation threshold are classified as The background mask.
其中,所述前景掩模(Fg_mask)就表示一个标志图像,其中像素只有0和1,是1的像素表示该点属于前景,是0的像素表示该点属于背景;所述背景掩模(Bg_mask)就是前景掩模(Fg_mask)取反。Wherein, the foreground mask (Fg_mask) represents a logo image, in which pixels only have 0 and 1, a pixel of 1 indicates that the point belongs to the foreground, and a pixel of 0 indicates that the point belongs to the background; the background mask (Bg_mask) ) Is the inverse of the foreground mask (Fg_mask).
步骤S30、根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数。Step S30: Determine a foreground depth value parameter and a background depth value parameter according to the foreground mask and the background mask, respectively.
根据所述前景掩模和所述背景掩模分别统计前景深度值均值和前景深度值均方差,以及背景深度值最大值和背景深度值最小值;即所述前景深度值参数包括前景深度值均值(Fg_mean)和前景深度值均方差(Fg_std),所述背景深度值参数包括背景深度值最大值(Bg_max)和背景深度值最小值(Bg_min)。According to the foreground mask and the background mask, the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are counted respectively; that is, the foreground depth value parameter includes the mean value of the foreground depth value (Fg_mean) and the mean square error of the foreground depth value (Fg_std). The background depth value parameter includes a maximum background depth value (Bg_max) and a minimum background depth value (Bg_min).
进一步地,依据所述前景掩模(Fg_mask)中标志位为1的像素点位置可以得到对应深度图像素,统计深度图中所有前景像素点的像素值,求均值,求方差,所述背景掩模(Bg_mask)在得到深度图中所有背景像素点的像素值之后,即可得到背景深度值最大值和背景深度值最小值。Further, the corresponding depth map pixel can be obtained according to the position of the pixel with the flag of 1 in the foreground mask (Fg_mask), the pixel values of all the foreground pixels in the depth map are counted, the average value is calculated, and the variance is calculated, and the background mask After obtaining the pixel values of all background pixels in the depth map, the mask (Bg_mask) can obtain the maximum background depth value and the minimum background depth value.
步骤S40、根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,并根据所述模糊核大小对所述第一图片进行背景虚化。Step S40: Determine a blur kernel size for blurring according to the foreground depth value parameter and the background depth value parameter, and perform background blurring on the first picture according to the blur kernel size.
其中,所述背景虚化为逐像素的背景虚化。Wherein, the background blur is a pixel-by-pixel background blur.
确定粗前景和粗背景虚化的级数,粗前景和前景的绝大部分是重叠的,可能在边缘部分不重叠,比较粗糙,前景比粗前景更加精确,粗背景和背景的关系也是一样的;以及确定多级虚化核的类型以及核大小,虚化核的大小不一致称为多级的虚化核,比如3x3、5x5、7x7等不同尺寸的正方形核。Determine the blur level of the rough foreground and the rough background. Most of the rough foreground and the foreground overlap, and may not overlap at the edge. The foreground is more accurate than the rough foreground. The relationship between the rough background and the background is also the same. ; And determine the type and size of the multi-level virtual core, the size of the virtual core is inconsistent called multi-level virtual core, such as 3x3, 5x5, 7x7 and other square cores of different sizes.
进一步地,所述根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小具体包括:Further, the determining the size of the blur kernel used for blurring according to the foreground depth value parameter and the background depth value parameter specifically includes:
(1)确定前景的虚化核为高斯核(高斯核就是核内的权重值服从二维的离散高斯分布),则当前像素的虚化半径Fg_r为:(1) Determine that the blur kernel of the foreground is a Gaussian kernel (the Gaussian kernel means that the weight value in the kernel obeys a two-dimensional discrete Gaussian distribution), then the blur radius Fg_r of the current pixel is:
其中,Fg_level为虚化级数,depthVal为深度值,Fg_mean为前景深度值均值,Fg_std为前景深度值均方差;Fg_level为2;Among them, Fg_level is the blur level, depthVal is the depth value, Fg_mean is the mean value of the foreground depth value, Fg_std is the mean square error of the foreground depth value; Fg_level is 2;
模糊核(模糊核就是一个带权重值的正方形)的大小为(2r+1)*(2r+1),其中各权重(各权重表示一个核内的各个不同权重值,权重就是一个数值,反应某个像素点的重要程度)的计算公式如下:The size of the fuzzy kernel (a fuzzy kernel is a square with a weight value) is (2r+1)*(2r+1), where each weight (each weight represents a different weight value in a kernel, the weight is a value, the reaction The calculation formula of the importance of a certain pixel is as follows:
Fg_k=exp(-Fg_r*dist);Fg_k=exp(-Fg_r*dist);
其中,dist为核中点(r,r)与其邻域的欧氏距离;Among them, dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;
(2)确定背景的虚化核为散焦模糊核(散焦模糊核就是在一个正方形的核内,其内切圆内的权重值不为0,内切圆以外的权重值都是0),则当前像素的虚化半径Bg_r为:(2) Determine that the background blur kernel is a defocus blur kernel (the defocus blur kernel is in a square kernel, the weight value in the inscribed circle is not 0, and the weight value outside the inscribed circle is 0) , The blur radius Bg_r of the current pixel is:
Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);
其中,Bg_level为虚化级数,Bg_max为背景深度值最大值,Bg_min为背景深度值最小值;Bg_level为11;Among them, Bg_level is the number of blur levels, Bg_max is the maximum background depth value, Bg_min is the minimum background depth value; Bg_level is 11;
模糊核(模糊核就是一个带权重值的正方形)的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel (a fuzzy kernel is a square with a weight value) is (2r+1)*(2r+1), and the calculation formula for each weight is as follows:
Bg_k=a*dist+b;Bg_k=a*dist+b;
其中,a和b表示权重值,用于控制dist值的影响大小,且0<a<1,b>1,a=0.1,b=10。Among them, a and b represent weight values, which are used to control the influence of the dist value, and 0<a<1, b>1, a=0.1, b=10.
对所述移动终端的主摄像头拍摄的第一图片(副摄像头的图片不用处理,副摄像头的作用是用于配合主摄像头生成深度图用的)依据对应的深度图遍历做逐像素的虚化处理,每个像素的模糊核取决于对应的深度值depthVal,模糊核选取如上所述。The first picture taken by the main camera of the mobile terminal (the picture of the secondary camera does not need to be processed, and the function of the secondary camera is to cooperate with the main camera to generate a depth map), and perform pixel-by-pixel blurring according to the corresponding depth map traversal , The blur kernel of each pixel depends on the corresponding depth value depthVal, and the blur kernel is selected as described above.
进一步地,因为逐像素的遍历比较耗时,各像素之间的虚化操作是独立的,可以采用多线程并行加速计算得到背景虚化效果图,如图4所示。本公开可以改善移动终端双摄像头拍照背景虚化的视觉效果,减少移动终端的内存消耗。Further, because the pixel-by-pixel traversal is relatively time-consuming, the blur operation between pixels is independent, and multi-threaded parallel acceleration calculation can be used to obtain the background blur effect map, as shown in FIG. 4. The present disclosure can improve the visual effect of the blurred background of the dual-camera photography of the mobile terminal, and reduce the memory consumption of the mobile terminal.
本公开直接在原图(例如第一图片和第二图片)上做基于不同模糊核大小的模糊处理,前景区域的深度较小,模糊核大小基本为1,本质上等价于不做虚化;即使存在异常深度值,做一个小模糊核(3x3)的模糊处理,也不影响最终的视觉效果;不做背景的分层多级虚化处理叠加,避免前景与背景叠加的边缘出现光晕或过渡不自然的现象,实现实时多级背景虚化效果。The present disclosure directly performs blur processing based on different blur kernel sizes on the original images (such as the first picture and the second picture), the depth of the foreground area is small, and the blur kernel size is basically 1, which is essentially equivalent to no blurring; Even if there is an abnormal depth value, a small blur core (3x3) will not affect the final visual effect; do not do the layered multi-level blur processing superimposition of the background, avoiding the halo or the edge of the foreground and the background superimposed Transition unnatural phenomenon, real-time multi-level background blur effect.
进一步地,发明人经过研究发现,在智能手机拍照领域,现有的对拍摄照片处理中背景虚化主要是利用主摄像头和副摄像头合成的深度图,从深度图中分割出前景和背景,然后将分层多级背景虚化后的背景与前景叠加生成背景虚化效果,然而利用深度图的分层多级背景虚化以及前景与背景的叠加都存在一定的缺陷,影响照片拍摄的效果,无法满足手机拍照的实时性。Further, the inventor found through research that in the field of smartphone photography, the existing background blurring in the processing of captured photos mainly uses the depth map synthesized by the main camera and the sub-camera to segment the foreground and background from the depth map, and then The background and foreground after layered multi-level background blur are superimposed to generate a background blur effect. However, the layered multi-level background blur using depth map and the superposition of foreground and background have certain defects, which affect the effect of photo shooting. Can not meet the real-time nature of mobile phone photography.
为了解决上述问题,在本公开实施例中,主要通过三个方面提升背景虚化的效果:1、通过背景图片的像素的拉伸操作和反拉伸操作,使得虚化后的背景具有一定的光斑效果;2、以前景的轮廓质心为圆心,以圆形向外扩散增大模糊核半径的方式进行逐点虚化,达到了多级虚化但不用分层的目的,解决多层分级虚化背景叠加的边缘过渡不自然问题;3、对前景的掩模进行腐蚀和高斯模糊平滑处理的方式,生成前景与背景过渡带,融合过渡带的前景和背景,有效缓解边缘过渡突兀,出现黑边和光晕等问题。In order to solve the above-mentioned problems, in the embodiments of the present disclosure, the effect of background blur is mainly improved in three aspects: 1. Through the stretching operation and reverse stretching operation of the pixels of the background picture, the blurred background has a certain degree of Spot effect; 2. Taking the center of the foreground contour as the center of the circle, the circle is diffused outward to increase the radius of the blur core to blur point by point, achieving the purpose of multi-level blur but not layering, and solves the multi-level blur Improve the unnatural problem of edge transition of background superimposition; 3. The method of etching and Gaussian blur smoothing of the foreground mask generates a transition zone between the foreground and the background, and merges the foreground and background of the transition zone, effectively alleviating the abrupt edge transition and the appearance of black Issues such as edges and halos.
下面结合附图,详细说明本公开的各种非限制性实施方式。Hereinafter, various non-limiting embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.
本公开实施例提供了一种拍照背景虚化方法中第二种虚化方式,如图5所示,所述方法包括:The embodiment of the present disclosure provides a second blurring method in the background blurring method for photographing. As shown in FIG. 5, the method includes:
S1、对深度图做预处理,得到前景图片和背景图片。S1. Preprocess the depth map to obtain the foreground image and the background image.
在本公开实施例中,在对深度图做预处理之前,获取拍摄图片和所述深度图。步骤S1中对深度图进行预处理,分割出前景图片F_Img和背景图片B_Img并得到相应的前景掩模F_mask和背景掩模B_mask,根据前景掩模F_mask和背景掩模B_mask得到前景图片和背景图片。其中,深度图可以是通过拍摄图片处理后得到的图片。前景图片可以为拍摄图片中的主体、距离镜头较近的物体、用户想 要拍摄的物体或者手机相机中焦点出的物体等图片,背景图片为图片中除前景区域外的图片。例如,对人进行拍摄,则图片中的人的区域为前景图片,其余区域为背景图片。In the embodiment of the present disclosure, before preprocessing the depth map, the captured picture and the depth map are acquired. In step S1, the depth map is preprocessed, the foreground image F_Img and the background image B_Img are segmented, and the corresponding foreground mask F_mask and background mask B_mask are obtained, and the foreground image and background image are obtained according to the foreground mask F_mask and the background mask B_mask. Among them, the depth map may be a picture obtained by taking a picture and processing it. The foreground picture can be the subject in the captured picture, the object close to the lens, the object that the user wants to take, or the focus of the mobile phone camera, etc. The background picture is the picture excluding the foreground area in the picture. For example, if a person is photographed, the area of the person in the picture is the foreground picture, and the rest of the area is the background picture.
在本公开一种可选实施例中,步骤S1包括:In an optional embodiment of the present disclosure, step S1 includes:
S11、对所述深度图进行保边滤波,得到第一深度图片。S11. Perform edge-preserving filtering on the depth image to obtain a first depth image.
S12、对所述第一深度图片进行中值滤波,得到第二深度图片。S12. Perform median filtering on the first depth picture to obtain a second depth picture.
S13、对所述第二深度图片进行水平集分割,得到前景掩模和背景掩模。S13. Perform level set segmentation on the second depth picture to obtain a foreground mask and a background mask.
在本公开实施例中,为了去除第一深度图片中的毛边,步骤S12中对第一深度图片进行中值滤波。通过对第二深度图片进行水平集分割,得到前景区域轮廓,根据前景区域轮廓得到前景掩模和背景掩模。第一深度图片和第二深度图片都是处理过程中的深度图片。In the embodiment of the present disclosure, in order to remove the burrs in the first depth picture, median filtering is performed on the first depth picture in step S12. By performing level set segmentation on the second depth image, the contour of the foreground area is obtained, and the foreground mask and the background mask are obtained according to the contour of the foreground area. The first depth picture and the second depth picture are both depth pictures during processing.
S14、根据所述前景掩模和所述背景掩模,得到所述前景图片和所述背景图片。S14. Obtain the foreground picture and the background picture according to the foreground mask and the background mask.
S2、对所述背景图片进行虚化,得到虚化背景图片。S2. Blur the background picture to obtain a blurred background picture.
在本公开实施例中,虚化背景图片为虚化后的背景图片。步骤S2中对背景图片中超过第一阈值threshold的像素值pixelValue进行拉伸变换,然后对拉伸后的背景图片做逐点的多级虚化,最后对虚化后的背景图片进行反拉伸变换,得到虚化背景图片。In the embodiment of the present disclosure, the blurred background picture is a blurred background picture. In step S2, the pixelValue in the background image that exceeds the first threshold is stretched and transformed, and then the stretched background image is subjected to point-by-point multi-level blurring, and finally the blurring background image is reverse-stretched Transform to get a blurred background image.
在本公开一种可选实施例中,步骤S2包括:In an optional embodiment of the present disclosure, step S2 includes:
S21、对所述背景图片进行拉伸变换,得到第一背景图片。S21: Perform stretching transformation on the background picture to obtain a first background picture.
在本公开实施例中,第一背景图片为拉伸变换后的背景图片。对超过第一阈值的背景图片中的像素值进行拉伸变换,其中,拉伸变换的表征形式如下:In the embodiment of the present disclosure, the first background picture is a background picture after stretching and transformation. Perform stretching transformation on the pixel values in the background image that exceed the first threshold, where the representation form of the stretching transformation is as follows:
pixelValue=a*(pixelValue-threshold)
b+threshold
pixelValue=a*(pixelValue-threshold) b +threshold
其中:threshold为第一阈值,a、b和threshold的值依赖具体要达到的光斑效果而定,其中,b>1且a>1,例如:a=2,b=2,threshold=128。Wherein: threshold is the first threshold, the values of a, b, and threshold depend on the specific spot effect to be achieved, where b>1 and a>1, for example: a=2, b=2, and threshold=128.
S22、对所述第一背景图片进行逐点多级虚化,得到第二背景图片。S22. Perform point-by-point multi-level blurring on the first background picture to obtain a second background picture.
在本公开实施例中,第二背景图片为逐点多级虚化后的背景图片。根据前景的轮廓计算出前景轮廓的质心点,计算出拉伸后的背景图片中每一个像素点与质心点的直线距离。In the embodiment of the present disclosure, the second background picture is a background picture that is blurred by point-by-point multi-level blur. According to the foreground contour, the centroid point of the foreground contour is calculated, and the linear distance between each pixel point and the centroid point in the stretched background image is calculated.
本公开实施例中通过逐点多级虚化的方法解决了现有技术中利用深度距离信息的分层多级虚化会使得相邻的区域出现虚化等级不一致、虚化分级较多会导致处理时间太长,无法满足手机端的实时性等问题,使背景的虚化程度大而且过渡比较自然。In the embodiments of the present disclosure, a point-by-point multi-level blurring method is used to solve the problem that the hierarchical multi-level blurring using depth distance information in the prior art will cause inconsistent blur levels in adjacent areas, and more blur levels will cause The processing time is too long to meet the real-time performance of the mobile phone, which makes the background blurry and the transition more natural.
在本公开一种可选实施例中,步骤S22包括:In an optional embodiment of the present disclosure, step S22 includes:
S221、根据所述前景图片,计算所述前景图片的质心点的坐标。S221. Calculate the coordinates of the centroid point of the foreground picture according to the foreground picture.
在本公开实施例中,可以根据前景图片中前景的轮廓计算出前景轮廓的质心点(x0,y0)。In the embodiment of the present disclosure, the centroid point (x0, y0) of the foreground contour can be calculated according to the contour of the foreground in the foreground picture.
S222、根据所述第一背景图片中像素点的坐标和所述质心点的坐标,计算所述第一背景图片中每一个像素点与质心点之间的直线距离。S222: Calculate the linear distance between each pixel point in the first background picture and the centroid point according to the coordinates of the pixel points in the first background picture and the coordinates of the centroid point.
在本公开实施例中,第一背景图片为拉伸后的背景图片,根据所述第一背景图片中像素点的坐标(i,j)和所述质心点的坐标(x0,y0),计算所述第一背景图片中每一个像素点(i,j)与质心点(x0,y0)之间的直线距离。In the embodiment of the present disclosure, the first background picture is a stretched background picture, and it is calculated according to the coordinates (i, j) of the pixel points in the first background picture and the coordinates (x0, y0) of the centroid point The linear distance between each pixel point (i, j) and the centroid point (x0, y0) in the first background picture.
S223、根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径。S223: Calculate the blur radius of the pixel in the first background picture according to the straight line distance and the blur degree value.
在本公开实施例中,根据前景的轮廓计算出前景轮廓的质心点(x0,y0),计算出所有(i,j)与(x0,y0)之间的直线距离dist。像素点虚化的半径的表征形式为:In the embodiment of the present disclosure, the centroid point (x0, y0) of the foreground contour is calculated according to the contour of the foreground, and the linear distance dist between all (i, j) and (x0, y0) is calculated. The characterization form of the radius of the pixel point blur is:
radius=baseRadius*(dist
i,j/max Dist)
radius=baseRadius*(dist i, j /max Dist)
其中,disti,j是背景图像坐标(i,j)的像素点与(x0,y0)的距离,maxDist为所有直线距离中最大值,baseRadius为虚化程度值,baseRadius根据要虚化的程度取值,例如:baseRadius可以为15。Among them, disti, j is the distance between the pixel point of the background image coordinate (i, j) and (x0, y0), maxDist is the maximum value of all straight-line distances, baseRadius is the value of the degree of blur, baseRadius is taken according to the degree of blur Value, for example: baseRadius can be 15.
在本公开一种可选实施例中,步骤S223之后,所述方法还包括:In an optional embodiment of the present disclosure, after step S223, the method further includes:
S224、根据所述像素点的虚化半径生成所述像素点的模糊核。S224: Generate a blur kernel of the pixel point according to the blur radius of the pixel point.
S225、根据所述像素点的模糊核,对所述像素点做模糊处理。S225: Perform blur processing on the pixel according to the blur kernel of the pixel.
在本公开实施例中,遍历第一背景图片中的像素点,依据像素点的虚化半径生成对应的模糊核,对像素点做模糊处理。逐像素的遍历比较耗时,各像素之间的虚化操作是独立的,可以采用多线程并行加速处理。In the embodiment of the present disclosure, the pixels in the first background picture are traversed, the corresponding blur kernel is generated according to the blur radius of the pixel, and the pixel is blurred. The pixel-by-pixel traversal is relatively time-consuming, and the blur operation between pixels is independent, and multi-threaded parallel processing can be used to accelerate processing.
S23、对所述第二背景图片做反拉伸变换,得到所述虚化背景图片。S23. Perform inverse stretch transformation on the second background picture to obtain the blurred background picture.
在本公开实施例中,第二背景图片为逐点多级虚化后的背景图片。反拉伸变换的表征形式为上述拉伸变换表达式的反函数。In the embodiment of the present disclosure, the second background picture is a background picture that is blurred by point-by-point multi-level blur. The characterization form of the inverse stretch transformation is the inverse function of the above-mentioned stretch transformation expression.
S3、对所述前景图片进行模糊,得到所述前景图片和所述背景图片之间的过渡带。S3. Blur the foreground picture to obtain a transition zone between the foreground picture and the background picture.
在本公开实施例中,对前景区域的掩模F_mask做腐蚀和高斯模糊处理,在掩模的边缘处产生像素值介于0~255(不包括0和255)的过渡带并作归一化到0~1,其中,过渡带就是前景与背景相邻的边缘区域,二值掩模图像的前景像素值为255,背景像素值为0,在前景与背景相邻的边缘部分,做一个平滑处理,例如高斯模糊处理,使得前景与背景边缘部分像素值介于0到255之间。这部分像素值在0~255之间的区域称为过渡带。In the embodiment of the present disclosure, the mask F_mask of the foreground area is processed by etching and Gaussian blur, and a transition zone with pixel values ranging from 0 to 255 (not including 0 and 255) is generated at the edge of the mask and normalized To 0~1, the transition zone is the edge area adjacent to the foreground and the background. The foreground pixel value of the binary mask image is 255, and the background pixel value is 0. A smoothing is performed on the edge part adjacent to the foreground and background. Processing, such as Gaussian blur processing, makes the pixel values of the foreground and background edge parts between 0 and 255. The area where the pixel value is between 0 and 255 is called the transition zone.
S4、将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片。S4. Superimpose the foreground picture and the blurred background picture and merge them in the transition zone to obtain a picture with a blurred background.
在本公开一种可选实施例中,步骤S4包括:In an optional embodiment of the present disclosure, step S4 includes:
在非过渡带区域,用所述前景图片或者所述虚化背景图片中的像素点对拍摄图片中对应的像素点进行替换;In the non-transition zone area, replace the corresponding pixels in the captured picture with the pixels in the foreground picture or the blurred background picture;
例如:在拍摄图片非过渡带区域中属于前景图片的像素点,则用该前景图片中对应位置的像素点进行替换,在拍摄图片非过渡带区域中属于背景图片的像素点,则用该背景图片中对应位置的像素点进行替换。替换完成后,非过渡带区域的图片的背景为虚化后的背景。For example: in the non-transition zone area of the taken picture, the pixels belonging to the foreground picture are replaced with the pixels of the corresponding position in the foreground picture, and the pixels belonging to the background picture in the non-transition zone area of the taken picture are replaced with the background The pixels at the corresponding positions in the picture are replaced. After the replacement is completed, the background of the picture in the non-transition zone is the blurred background.
在过渡带区域,根据模糊值、前景图片的像素值和虚化背景图片的像素值,得到过渡带区域融合后的像素值,其中,所述模糊值为对所述前景图片腐蚀和模糊后的像素值。In the transition zone area, according to the blur value, the pixel value of the foreground picture, and the pixel value of the blurred background picture, the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
在本公开实施例中,过渡带区域的像素点在步骤S3中处理的F_mask的像素值为w,该过渡带区域的像素点在前景图片中像素值为F_pixel,该过渡带区域的像素点在虚化背景图片中像素值为B_pixel,则该像素点在过渡带融合后的像素值为FB_pixel=W*F_pixel+(1-w)*B_pixel。In the embodiment of the present disclosure, the pixel value of the F_mask processed in step S3 of the pixel of the transition zone area is w, the pixel value of the transition zone area is F_pixel in the foreground image, and the pixel point of the transition zone area is The pixel value of the blurred background image is B_pixel, and the pixel value of the pixel after fusion in the transition zone is FB_pixel=W*F_pixel+(1-w)*B_pixel.
本公开实施例中通过在前景和背景之间生成过渡带的方式,对处理后的前景和背景进行融合,缓解了边缘过渡突兀,避免了前景和背景的边界出现黑边和光晕的问题。In the embodiment of the present disclosure, the processed foreground and background are merged by generating a transition zone between the foreground and the background, so as to alleviate the abrupt edge transition and avoid the problem of black borders and halos appearing on the boundary between the foreground and the background.
进一步地,如图6所示,基于上述拍照背景虚化方法,本公开还相应提供了一种移动终端,所述移动终端包括处理器10、存储器20及显示器30。图6仅示出了移动终端的部分组件,但是应理解的是,并不要求实施所有示出的组件,可以替代的实施更多或者更少的组件。Further, as shown in FIG. 6, based on the aforementioned method for blurring the background of a photograph, the present disclosure also provides a mobile terminal correspondingly, and the mobile terminal includes a processor 10, a memory 20 and a display 30. FIG. 6 only shows part of the components of the mobile terminal, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.
所述移动终端还包括第一摄像头组和第二摄像头组;所述第一摄像头组和所述第二摄像头组用于获取两张不同角度拍摄的第一图片和第二图片;所述第一图片由所述第一摄像头组拍摄得到的图片,所述第一摄像头组包括一个或者多个摄像头;所述第二图片由所述第二摄像头组拍摄得到的图片,所述第二摄像头组包括一个或者多个摄像头;所述第一摄像头组的摄像头与所述第二摄像头组的摄像头之间至少有一个不相同。The mobile terminal also includes a first camera group and a second camera group; the first camera group and the second camera group are used to acquire two first pictures and second pictures taken from different angles; the first The picture is a picture taken by the first camera group, the first camera group includes one or more cameras; the second picture is a picture taken by the second camera group, the second camera group includes One or more cameras; at least one of the cameras of the first camera group is different from the cameras of the second camera group.
其中,所述存储器20在一些实施例中可以是所述移动终端的内部存储单元,例如移动终端的硬盘或内存。所述存储器20在另一些实施例中也可以是所述移动终端的外部存储设备,例如所述移动终端上配备的插接式硬盘,智能存储卡(Smart Media Card,SMC),安全数字(Secure Digital,SD)卡,闪存卡(Flash Card)等。进一步地,所述存储器20还可以既包括所述移动终端的内部存储单元也包括外部存储设备。所述存储器20用于存储安装于所述移动终端的应用软件及各类数据,例如所述安装移动终端的程序代码等。所述存储器20还可以用于暂时地存储已经输出或者将要输出的数据。在一实施例中,存储器20上存储有拍照背景虚化程序40,该拍照背景虚化程序40可被处理器10所执行,从而实现本公开中拍照背景虚化方法。Wherein, the memory 20 may be an internal storage unit of the mobile terminal in some embodiments, such as a hard disk or a memory of the mobile terminal. In other embodiments, the memory 20 may also be an external storage device of the mobile terminal, such as a plug-in hard disk equipped on the mobile terminal, a smart media card (SMC), and a secure digital (Secure Digital). Digital, SD) card, flash card, etc. Further, the memory 20 may also include both an internal storage unit of the mobile terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the mobile terminal, such as the program code of the installed mobile terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a photographing background blurring program 40, and the photographing background blurring program 40 can be executed by the processor 10, so as to realize the photographing background blurring method in the present disclosure.
所述处理器10在一些实施例中可以是一中央处理器(Central Processing Unit,CPU),微处理器或其他数据处理芯片,用于运行所述存储器20中存储的程序代码或处理数据,例如执行所述拍照背景虚化方法等。The processor 10 may be a central processing unit (CPU), microprocessor or other data processing chip in some embodiments, and is used to run the program code or process data stored in the memory 20, for example Perform the method of blurring the background of the photograph and so on.
所述显示器30在一些实施例中可以是LED显示器、液晶显示器、触控式液晶显示器以及OLED(Organic Light-Emitting Diode,有机发光二极管)触摸器等。所述显示器30用于显示在所述移动终端的信息以及用于显示可视化的用户界面。所述移动终端的部件10-30通过***总线相互通信。In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. The display 30 is used for displaying information on the mobile terminal and for displaying a visualized user interface. The components 10-30 of the mobile terminal communicate with each other via a system bus.
在一实施例中,当处理器10执行所述存储器20中拍照背景虚化程序40时实现以下步骤:In one embodiment, when the processor 10 executes the photographing background blurring program 40 in the memory 20, the following steps are implemented:
获取深度图,将所述深度图进行预处理和分割处理;Acquire a depth map, and perform preprocessing and segmentation processing on the depth map;
将分割后的图片进行虚化处理,得到背景虚化后的图片。The segmented picture is blurred to obtain a picture with a blurred background.
所述获取深度图包括:The acquiring depth map includes:
获取第一图片和第二图片;Obtain the first picture and the second picture;
将第一图片和第二图片合成深度图。The first picture and the second picture are combined into a depth map.
所述将所述深度图进行预处理和分割处理包括:The preprocessing and segmentation processing of the depth map includes:
将所述深度图进行预处理,将预处理后的所述深度图进行分割得到前景掩模和背景掩模。The depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
所述将分割后的图片进行虚化处理包括:The blurring processing of the divided picture includes:
根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数;Respectively determining a foreground depth value parameter and a background depth value parameter according to the foreground mask and the background mask;
根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,并根据所述模糊核大小对所述第一图片进行背景虚化。The size of the blur kernel used for blurring is determined according to the foreground depth value parameter and the background depth value parameter, and the background blurring is performed on the first picture according to the blur kernel size.
所述第一图片是由移动终端的第一摄像头组拍摄得到的图片,所述第一摄像头组包括一个或者多个摄像头;The first picture is a picture taken by a first camera group of the mobile terminal, and the first camera group includes one or more cameras;
所述第二图片是由移动终端的第二摄像头组拍摄得到的图片,所述第二摄像头组包括一个或者多个摄像头;The second picture is a picture taken by a second camera group of the mobile terminal, and the second camera group includes one or more cameras;
所述第一摄像头组的摄像头与所述第二摄像头组的摄像头之间至少有一个不相同。At least one of the cameras of the first camera group is different from the cameras of the second camera group.
所述预处理包括:The preprocessing includes:
将所述深度图进行保边滤波处理;Performing edge-preserving filtering processing on the depth map;
将进行保边滤波处理后的所述深度图进行中值滤波操作。The depth map after edge-preserving filtering processing is subjected to a median filtering operation.
所述将预处理后的所述深度图进行分割得到前景掩模和背景掩模,具体包括:The segmentation of the preprocessed depth map to obtain a foreground mask and a background mask specifically includes:
将进行中值滤波操作后的所述深度图进行分割处理;Performing segmentation processing on the depth map after the median filtering operation;
根据预先设置的分割阈值得到所述前景掩模和所述背景掩模。The foreground mask and the background mask are obtained according to a preset segmentation threshold.
所述根据预先设置的分割阈值得到所述前景掩模和所述背景掩模,具体包括:The obtaining the foreground mask and the background mask according to a preset segmentation threshold specifically includes:
将所述深度图中像素值大于所述分割阈值的像素点归为所述前景掩模;Classify pixels with pixel values greater than the segmentation threshold in the depth map as the foreground mask;
将所述深度图中像素值小于或等于所述分割阈值的像素点归为所述背景掩模。The pixel points in the depth map whose pixel values are less than or equal to the segmentation threshold are classified as the background mask.
所述根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数,具体包括:The respectively determining the foreground depth value parameter and the background depth value parameter according to the foreground mask and the background mask specifically includes:
根据所述前景掩模和所述背景掩模分别统计前景深度值均值和前景深度值均方差,以及背景深度值最大值和背景深度值最小值。According to the foreground mask and the background mask, the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
所述根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,具体包括:The determining the size of the blur kernel used for blurring according to the foreground depth value parameter and the background depth value parameter specifically includes:
确定前景的虚化核为高斯核,则当前像素的虚化半径Fg_r为:It is determined that the blur kernel of the foreground is a Gaussian kernel, then the blur radius Fg_r of the current pixel is:
其中,Fg_level为虚化级数,depthVal为深度值,Fg_mean为前景深度值均值,Fg_std为前景深度值均方差;Among them, Fg_level is the blur level, depthVal is the depth value, Fg_mean is the mean value of the foreground depth value, and Fg_std is the mean square error of the foreground depth value;
模糊核的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
Fg_k=exp(-Fg_r*dist);Fg_k=exp(-Fg_r*dist);
其中,dist为核中点(r,r)与其邻域的欧氏距离;Among them, dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;
确定背景的虚化核为散焦模糊核,则当前像素的虚化半径Bg_r为:It is determined that the blur kernel of the background is a defocus blur kernel, then the blur radius Bg_r of the current pixel is:
Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);
其中,Bg_level为虚化级数,Bg_max为背景深度值最大值,Bg_min为背景深度值最小值;Among them, Bg_level is the number of blur levels, Bg_max is the maximum background depth value, and Bg_min is the minimum background depth value;
模糊核的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:
Bg_k=a*dist+b;Bg_k=a*dist+b;
其中,a和b表示权重值,且0<a<1,b>1。Among them, a and b represent weight values, and 0<a<1,b>1.
其中,所述Fg_level为2,所述Bg_level为11,a=0.1,b=10。Wherein, the Fg_level is 2, the Bg_level is 11, a=0.1, and b=10.
其中,所述背景虚化为逐像素的背景虚化。Wherein, the background blur is a pixel-by-pixel background blur.
所述拍照背景虚化方法还包括:The photo background blurring method further includes:
对深度图做预处理,得到前景图片和背景图片;Preprocess the depth map to get the foreground image and background image;
对所述背景图片进行虚化,得到虚化背景图片;Blur the background picture to obtain a blurred background picture;
对所述前景图片进行模糊,得到所述前景图片和所述背景图片之间的过渡带;Blur the foreground picture to obtain a transition zone between the foreground picture and the background picture;
将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片。The foreground picture and the blurred background picture are superimposed and merged in the transition zone to obtain a picture with a blurred background.
所述对深度图做预处理,得到前景图片和背景图片,包括:The preprocessing of the depth map to obtain the foreground image and the background image includes:
对所述深度图进行保边滤波,得到第一深度图片;Performing edge-preserving filtering on the depth image to obtain a first depth image;
对所述第一深度图片进行中值滤波,得到第二深度图片;Performing median filtering on the first depth picture to obtain a second depth picture;
对所述第二深度图片进行水平集分割,得到前景掩模和背景掩模;Performing level set segmentation on the second depth picture to obtain a foreground mask and a background mask;
根据所述前景掩模和所述背景掩模,得到所述前景图片和所述背景图片。According to the foreground mask and the background mask, the foreground picture and the background picture are obtained.
所述对所述背景图片进行虚化,得到虚化背景图片,包括:The blurring the background picture to obtain a blurring background picture includes:
对所述背景图片进行拉伸变换,得到第一背景图片;Stretch and transform the background picture to obtain a first background picture;
对所述第一背景图片进行逐点多级虚化,得到第二背景图片;Performing point-by-point multi-level blurring on the first background picture to obtain a second background picture;
对所述第二背景图片做反拉伸变换,得到所述虚化背景图片。Performing an inverse stretch transformation on the second background picture to obtain the blurred background picture.
所述对所述第一背景图片进行逐点多级虚化,得到第二背景图片,包括:The step-by-point multi-level blurring of the first background picture to obtain the second background picture includes:
根据所述前景图片,计算所述前景图片的质心点的坐标;Calculating the coordinates of the centroid point of the foreground picture according to the foreground picture;
根据所述第一背景图片中像素点的坐标和所述质心点的坐标,计算所述第一背景图片中每一个像素点与质心点之间的直线距离;Calculating the linear distance between each pixel point in the first background picture and the centroid point according to the coordinates of the pixel points in the first background picture and the coordinates of the centroid point;
根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径。According to the straight line distance and the blur degree value, the blur radius of the pixel in the first background picture is calculated.
根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径之后,包括:After calculating the blur radius of the pixel in the first background picture according to the straight line distance and the blur degree value, the method includes:
根据所述像素点的虚化半径生成所述像素点的模糊核;Generating a blur kernel of the pixel according to the blur radius of the pixel;
根据所述像素点的模糊核,对所述像素点做模糊处理。According to the blur kernel of the pixel point, blur processing is performed on the pixel point.
所述将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片,包括:The superimposing the foreground picture and the blurred background picture and fusing them in the transition zone to obtain a blurred background picture includes:
在非过渡带区域,用所述前景图片或者所述虚化背景图片中的像素点对拍摄图片中对应的像素点进行替换;In the non-transition zone area, replace the corresponding pixels in the captured picture with the pixels in the foreground picture or the blurred background picture;
在过渡带区域,根据模糊值、前景图片的像素值和虚化背景图片的像素值,得到过渡带区域融合后的像素值,其中,所述模糊值为对所述前景图片腐蚀和模糊后的像素值。In the transition zone area, according to the blur value, the pixel value of the foreground picture, and the pixel value of the blurred background picture, the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
所述对深度图做预处理,得到前景图片和背景图片之前,包括:The preprocessing of the depth map to obtain the foreground image and the background image includes:
获取拍摄图片和所述深度图。Obtain the photographed picture and the depth map.
本公开还提供一种存储介质,其中,所述存储介质存储有拍照背景虚化程序,所述拍照背景虚化程序被处理器执行时实现如上所述的拍照背景虚化方法的步骤。The present disclosure also provides a storage medium, wherein the storage medium stores a photographing background blurring program, and the photographing background blurring program is executed by a processor to realize the steps of the photographing background blurring method as described above.
综上所述,本公开提供一种拍照背景虚化方法、移动终端及存储介质,所述方法包括:获取深度图,将所述深度图进行预处理和分割处理;将分割后的图片进行虚化处理,得到背景虚化后的图片。本公开对拍摄的图片进行处理,可以使拍摄的照片具有背景多级虚化的效果,提升背景虚化的效果。In summary, the present disclosure provides a method for blurring a photographed background, a mobile terminal, and a storage medium. The method includes: acquiring a depth map, performing preprocessing and segmentation processing on the depth map; and performing a virtual image on the segmented picture. After the bokeh process, the picture with the blurred background is obtained. The present disclosure processes the taken pictures, so that the taken pictures have the effect of multi-level blurring of the background, and the effect of the background blurring can be improved.
当然,本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关硬件(如处理器,控制器等)来完成,所述的程序可存储于一计算机可读取的存储介质中,所述程序在执行时可包括如上述各方法实施例的流程。其中所述的存储介质可为存储器、磁碟、光盘等。Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be implemented by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer program. In a computer-readable storage medium, the program may include the processes of the foregoing method embodiments when executed. The storage medium mentioned may be a memory, a magnetic disk, an optical disk, and the like.
应当理解的是,本公开的应用不限于上述的举例,对本领域普通技术人员来说,可以根据上述说明加以改进或变换,所有这些改进和变换都应属于本公开所附权利要求的保护范围。It should be understood that the application of the present disclosure is not limited to the above examples, and those of ordinary skill in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the protection scope of the appended claims of the present disclosure.
Claims (21)
- 一种拍照背景虚化方法,其中,所述拍照背景虚化方法包括:A method for blurring a photographed background, wherein the method for blurring a photographed background includes:获取深度图,将所述深度图进行预处理和分割处理;Acquire a depth map, and perform preprocessing and segmentation processing on the depth map;将分割后的图片进行虚化处理,得到背景虚化后的图片。The segmented picture is blurred to obtain a picture with a blurred background.
- 根据权利要求1所述的拍照背景虚化方法,其中,所述获取深度图包括:The method for blurring a photographed background according to claim 1, wherein said acquiring a depth map comprises:获取第一图片和第二图片;Obtain the first picture and the second picture;将第一图片和第二图片合成深度图。The first picture and the second picture are combined into a depth map.
- 根据权利要求1所述的拍照背景虚化方法,其中,所述将所述深度图进行预处理和分割处理包括:The method for blurring a photographed background according to claim 1, wherein the preprocessing and segmentation processing of the depth map comprises:将所述深度图进行预处理,将预处理后的所述深度图进行分割得到前景掩模和背景掩模。The depth map is preprocessed, and the preprocessed depth map is segmented to obtain a foreground mask and a background mask.
- 根据权利要求3所述的拍照背景虚化方法,其中,所述将分割后的图片进行虚化处理包括:The method for blurring a photographed background according to claim 3, wherein said performing blurring processing on the segmented picture comprises:根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数;Respectively determining a foreground depth value parameter and a background depth value parameter according to the foreground mask and the background mask;根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,并根据所述模糊核大小对所述第一图片进行背景虚化。The size of the blur kernel used for blurring is determined according to the foreground depth value parameter and the background depth value parameter, and the background blurring is performed on the first picture according to the blur kernel size.
- 根据权利要求2所述的拍照背景虚化方法,其中,所述第一图片是由移动终端的第一摄像头组拍摄得到的图片,所述第一摄像头组包括一个或者多个摄像头;The method for blurring a photographic background according to claim 2, wherein the first picture is a picture taken by a first camera group of a mobile terminal, and the first camera group includes one or more cameras;所述第二图片是由移动终端的第二摄像头组拍摄得到的图片,所述第二摄像头组包括一个或者多个摄像头;The second picture is a picture taken by a second camera group of the mobile terminal, and the second camera group includes one or more cameras;所述第一摄像头组的摄像头与所述第二摄像头组的摄像头之间至少有一个不相同。At least one of the cameras of the first camera group is different from the cameras of the second camera group.
- 根据权利要求3所述的拍照背景虚化方法,其中,所述预处理包括:The method for blurring a photographed background according to claim 3, wherein the preprocessing comprises:将所述深度图进行保边滤波处理;Performing edge-preserving filtering processing on the depth map;将进行保边滤波处理后的所述深度图进行中值滤波操作。The depth map after edge-preserving filtering processing is subjected to a median filtering operation.
- 根据权利要求6所述的拍照背景虚化方法,其中,所述将预处理后的所述深度图进行分割得到前景掩模和背景掩模,具体包括:The method for blurring a photographed background according to claim 6, wherein the segmenting the preprocessed depth map to obtain a foreground mask and a background mask specifically includes:将进行中值滤波操作后的所述深度图进行分割处理;Performing segmentation processing on the depth map after the median filtering operation;根据预先设置的分割阈值得到所述前景掩模和所述背景掩模。The foreground mask and the background mask are obtained according to a preset segmentation threshold.
- 根据权利要求7所述的拍照背景虚化方法,其中,所述根据预先设置的分割阈值得到所述前景掩模和所述背景掩模,具体包括:8. The method for blurring a photographed background according to claim 7, wherein the obtaining the foreground mask and the background mask according to a preset segmentation threshold specifically includes:将所述深度图中像素值大于所述分割阈值的像素点归为所述前景掩模;Classify pixels with pixel values greater than the segmentation threshold in the depth map as the foreground mask;将所述深度图中像素值小于或等于所述分割阈值的像素点归为所述背景掩模。The pixel points in the depth map whose pixel values are less than or equal to the segmentation threshold are classified as the background mask.
- 根据权利要求4任一项所述的拍照背景虚化方法,其中,所述根据所述前景掩模和所述背景掩模分别确定前景深度值参数和背景深度值参数,具体包括:The method for blurring a photographed background according to any one of claims 4, wherein said determining a foreground depth value parameter and a background depth value parameter respectively according to the foreground mask and the background mask specifically includes:根据所述前景掩模和所述背景掩模分别统计前景深度值均值和前景深度值均方差,以及背景深度值最大值和背景深度值最小值。According to the foreground mask and the background mask, the mean value of the foreground depth value and the mean square error of the foreground depth value, as well as the maximum value of the background depth value and the minimum value of the background depth value are respectively counted.
- 根据权利要求9所述的拍照背景虚化方法,其中,所述根据所述前景深度值参数和所述背景深度值参数确定用于虚化的模糊核大小,具体包括:The method for blurring a photographed background according to claim 9, wherein the determining the size of the blur kernel used for blurring according to the foreground depth value parameter and the background depth value parameter specifically includes:确定前景的虚化核为高斯核,则当前像素的虚化半径Fg_r为:It is determined that the blur kernel of the foreground is a Gaussian kernel, then the blur radius Fg_r of the current pixel is:其中,Fg_level为虚化级数,depthVal为深度值,Fg_mean为前景深度值均值,Fg_std为前景深度值均方差;Among them, Fg_level is the blur level, depthVal is the depth value, Fg_mean is the mean value of the foreground depth value, and Fg_std is the mean square error of the foreground depth value;模糊核的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:Fg_k=exp(-Fg_r*dist);Fg_k=exp(-Fg_r*dist);其中,dist为核中点(r,r)与其邻域的欧氏距离;Among them, dist is the Euclidean distance between the kernel midpoint (r,r) and its neighbors;确定背景的虚化核为散焦模糊核,则当前像素的虚化半径Bg_r为:It is determined that the blur kernel of the background is a defocus blur kernel, then the blur radius Bg_r of the current pixel is:Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);Bg_r=Bg_level*(depthVal-Bg_min)/(Bg_max-Bg_min+1);其中,Bg_level为虚化级数,Bg_max为背景深度值最大值,Bg_min为背景深度值最小值;Among them, Bg_level is the number of blur levels, Bg_max is the maximum background depth value, and Bg_min is the minimum background depth value;模糊核的大小为(2r+1)*(2r+1),其中各权重的计算公式如下:The size of the fuzzy kernel is (2r+1)*(2r+1), and the calculation formula of each weight is as follows:Bg_k=a*dist+b;Bg_k=a*dist+b;其中,a和b表示权重值,且0<a<1,b>1。Among them, a and b represent weight values, and 0<a<1,b>1.
- 根据权利要求10所述的拍照背景虚化方法,其中,所述Fg_level为2,所述Bg_level为11,a=0.1,b=10。10. The method for blurring a photographic background according to claim 10, wherein the Fg_level is 2, the Bg_level is 11, a=0.1, and b=10.
- 根据权利要求1所述的拍照背景虚化方法,其中,所述背景虚化为逐像素的背景虚化。The method for blurring a photographed background according to claim 1, wherein the background blurring is a pixel-by-pixel background blurring.
- 根据权利要求1所述的拍照背景虚化方法,其中,所述拍照背景虚化方法还包括:The method for blurring a photographed background according to claim 1, wherein the method for blurring a photographed background further comprises:对深度图做预处理,得到前景图片和背景图片;Preprocess the depth map to get the foreground image and background image;对所述背景图片进行虚化,得到虚化背景图片;Blur the background picture to obtain a blurred background picture;对所述前景图片进行模糊,得到所述前景图片和所述背景图片之间的过渡带;Blur the foreground picture to obtain a transition zone between the foreground picture and the background picture;将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片。The foreground picture and the blurred background picture are superimposed and merged in the transition zone to obtain a picture with a blurred background.
- 根据权利要求13所述的方法,其中,所述对深度图做预处理,得到前景图片和背景图片,包括:The method according to claim 13, wherein said preprocessing the depth map to obtain the foreground picture and the background picture comprises:对所述深度图进行保边滤波,得到第一深度图片;Performing edge-preserving filtering on the depth image to obtain a first depth image;对所述第一深度图片进行中值滤波,得到第二深度图片;Performing median filtering on the first depth picture to obtain a second depth picture;对所述第二深度图片进行水平集分割,得到前景掩模和背景掩模;Performing level set segmentation on the second depth picture to obtain a foreground mask and a background mask;根据所述前景掩模和所述背景掩模,得到所述前景图片和所述背景图片。According to the foreground mask and the background mask, the foreground picture and the background picture are obtained.
- 根据权利要求13所述的方法,其中,所述对所述背景图片进行虚化,得到虚化背景图片,包括:The method according to claim 13, wherein said blurring said background picture to obtain a blurred background picture comprises:对所述背景图片进行拉伸变换,得到第一背景图片;Stretch and transform the background picture to obtain a first background picture;对所述第一背景图片进行逐点多级虚化,得到第二背景图片;Performing point-by-point multi-level blurring on the first background picture to obtain a second background picture;对所述第二背景图片做反拉伸变换,得到所述虚化背景图片。Performing an inverse stretch transformation on the second background picture to obtain the blurred background picture.
- 根据权利要求15所述的方法,其中,所述对所述第一背景图片进行逐点多级虚 化,得到第二背景图片,包括:The method according to claim 15, wherein said performing point-by-point multi-level blurring on the first background picture to obtain the second background picture comprises:根据所述前景图片,计算所述前景图片的质心点的坐标;Calculating the coordinates of the centroid point of the foreground picture according to the foreground picture;根据所述第一背景图片中像素点的坐标和所述质心点的坐标,计算所述第一背景图片中每一个像素点与质心点之间的直线距离;Calculating the linear distance between each pixel point in the first background picture and the centroid point according to the coordinates of the pixel points in the first background picture and the coordinates of the centroid point;根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径。According to the straight line distance and the blur degree value, the blur radius of the pixel in the first background picture is calculated.
- 根据权利要求16所述的方法,其中,根据所述直线距离和虚化程度值,计算所述第一背景图片中像素点的虚化半径之后,还包括:The method according to claim 16, wherein after calculating the blur radius of the pixel in the first background image according to the straight line distance and the blur degree value, the method further comprises:根据所述像素点的虚化半径生成所述像素点的模糊核;Generating a blur kernel of the pixel according to the blur radius of the pixel;根据所述像素点的模糊核,对所述像素点做模糊处理。According to the blur kernel of the pixel point, blur processing is performed on the pixel point.
- 根据权利要求13所述的方法,其中,所述将所述前景图片和所述虚化背景图片进行叠加并在所述过渡带进行融合,得到背景虚化后的图片,包括:The method according to claim 13, wherein the superimposing the foreground picture and the blurred background picture and fusing them in the transition zone to obtain a picture with a blurred background comprises:在非过渡带区域,用所述前景图片或者所述虚化背景图片中的像素点对拍摄图片中对应的像素点进行替换;In the non-transition zone area, replace the corresponding pixels in the captured picture with the pixels in the foreground picture or the blurred background picture;在过渡带区域,根据模糊值、前景图片的像素值和虚化背景图片的像素值,得到过渡带区域融合后的像素值,其中,所述模糊值为对所述前景图片腐蚀和模糊后的像素值。In the transition zone area, according to the blur value, the pixel value of the foreground picture, and the pixel value of the blurred background picture, the fused pixel value of the transition zone area is obtained, wherein the blur value is the result of the erosion and blurring of the foreground picture Pixel values.
- 根据权利要求13所述的方法,其中,所述对深度图做预处理,得到前景图片和背景图片之前,还包括:The method according to claim 13, wherein before said preprocessing the depth map to obtain the foreground picture and the background picture, the method further comprises:获取拍摄图片和所述深度图。Obtain the photographed picture and the depth map.
- 一种移动终端,其中,所述移动终端包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的拍照背景虚化程序,所述拍照背景虚化程序被所述处理器执行时实现如权利要求1-19任一项所述的拍照背景虚化方法的步骤。A mobile terminal, wherein the mobile terminal includes a memory, a processor, and a photographing background blurring program stored in the memory and running on the processor, and the photographing background blurring program is When the processor is executed, the steps of the method for blurring the photographic background according to any one of claims 1-19 are realized.
- 一种存储介质,其中,所述存储介质存储有拍照背景虚化程序,所述拍照背景虚化程序被处理器执行时实现如权利要求1-19任一项所述的拍照背景虚化方法的步骤。A storage medium, wherein the storage medium stores a photographing background blurring program, and when the photographing background blurring program is executed by a processor, the method for implementing the photographing background blurring method according to any one of claims 1-19 step.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911397607.7A CN113129207B (en) | 2019-12-30 | 2019-12-30 | Picture background blurring method and device, computer equipment and storage medium |
CN201911397607.7 | 2019-12-30 | ||
CN202010085333.4 | 2020-02-10 | ||
CN202010085333.4A CN113256482B (en) | 2020-02-10 | 2020-02-10 | Photographing background blurring method, mobile terminal and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021135676A1 true WO2021135676A1 (en) | 2021-07-08 |
Family
ID=76686430
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/128657 WO2021135676A1 (en) | 2019-12-30 | 2020-11-13 | Photographing background blurring method, mobile terminal, and storage medium |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2021135676A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230298191A1 (en) * | 2022-03-16 | 2023-09-21 | Toyota Research Institute, Inc. | Multi-camera cost volumes for self-supervised depth estimation |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102693527A (en) * | 2011-02-28 | 2012-09-26 | 索尼公司 | Method and apparatus for performing a blur rendering process on an image |
CN103945118A (en) * | 2014-03-14 | 2014-07-23 | 华为技术有限公司 | Picture blurring method and device and electronic equipment |
US20150002545A1 (en) * | 2013-06-28 | 2015-01-01 | Canon Kabushiki Kaisha | Variable blend width compositing |
CN107085825A (en) * | 2017-05-27 | 2017-08-22 | 成都通甲优博科技有限责任公司 | Image weakening method, device and electronic equipment |
CN109559272A (en) * | 2018-10-30 | 2019-04-02 | 深圳市商汤科技有限公司 | A kind of image processing method and device, electronic equipment, storage medium |
-
2020
- 2020-11-13 WO PCT/CN2020/128657 patent/WO2021135676A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102693527A (en) * | 2011-02-28 | 2012-09-26 | 索尼公司 | Method and apparatus for performing a blur rendering process on an image |
US20150002545A1 (en) * | 2013-06-28 | 2015-01-01 | Canon Kabushiki Kaisha | Variable blend width compositing |
CN103945118A (en) * | 2014-03-14 | 2014-07-23 | 华为技术有限公司 | Picture blurring method and device and electronic equipment |
CN107085825A (en) * | 2017-05-27 | 2017-08-22 | 成都通甲优博科技有限责任公司 | Image weakening method, device and electronic equipment |
CN109559272A (en) * | 2018-10-30 | 2019-04-02 | 深圳市商汤科技有限公司 | A kind of image processing method and device, electronic equipment, storage medium |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230298191A1 (en) * | 2022-03-16 | 2023-09-21 | Toyota Research Institute, Inc. | Multi-camera cost volumes for self-supervised depth estimation |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10872420B2 (en) | Electronic device and method for automatic human segmentation in image | |
US11250571B2 (en) | Robust use of semantic segmentation in shallow depth of field rendering | |
US11526995B2 (en) | Robust use of semantic segmentation for depth and disparity estimation | |
Xiao et al. | Fast image dehazing using guided joint bilateral filter | |
EP1969562B1 (en) | Edge-guided morphological closing in segmentation of video sequences | |
EP1969560B1 (en) | Edge-controlled morphological closing in segmentation of video sequences | |
KR101036787B1 (en) | Motion vector calculation method, hand-movement correction device using the method, imaging device, and motion picture generation device | |
US9305360B2 (en) | Method and apparatus for image enhancement and edge verification using at least one additional image | |
CN107507217B (en) | Method and device for making certificate photo and storage medium | |
US20090028432A1 (en) | Segmentation of Video Sequences | |
JP2002245471A (en) | Photograph finishing service for double print accompanied by second print corrected according to subject contents | |
US11238302B2 (en) | Method and an apparatus for performing object illumination manipulation on an image | |
JP2006119817A (en) | Image processor | |
CN110503704B (en) | Method and device for constructing three-dimensional graph and electronic equipment | |
Kinoshita et al. | Automatic exposure compensation using an image segmentation method for single-image-based multi-exposure fusion | |
CN113129207B (en) | Picture background blurring method and device, computer equipment and storage medium | |
Lao et al. | Flow-guided video inpainting with scene templates | |
WO2021135676A1 (en) | Photographing background blurring method, mobile terminal, and storage medium | |
CN113674303B (en) | Image processing method, device, electronic equipment and storage medium | |
CN113658197B (en) | Image processing method, device, electronic equipment and computer readable storage medium | |
WO2017128646A1 (en) | Image processing method and device | |
CN115546027B (en) | Image suture line determination method, device and storage medium | |
Ernst et al. | Check my chart: A robust color chart tracker for colorimetric camera calibration | |
US20220398704A1 (en) | Intelligent Portrait Photography Enhancement System | |
WO2022127491A1 (en) | Image processing method and device, and storage medium and terminal |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20910472 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20910472 Country of ref document: EP Kind code of ref document: A1 |