CN111754521B

CN111754521B - Image processing method and device, electronic equipment and storage medium

Info

Publication number: CN111754521B
Application number: CN202010554916.7A
Authority: CN
Inventors: 张弓
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2020-06-17
Filing date: 2020-06-17
Publication date: 2024-06-25
Anticipated expiration: 2040-06-17
Also published as: CN111754521A

Abstract

The application relates to an image processing method, comprising the following steps: acquiring an image to be processed and an image direction; adjusting the picture direction of the image to be processed according to the matching property of the image direction of the image to be processed and the target processing direction of the main body segmentation model; inputting the processed image to be processed into a main body segmentation model, wherein the main body segmentation model is obtained by training according to a training image and a corresponding labeled label main body mask image in advance, and the training process comprises image preprocessing for adjusting the training image and the corresponding labeled label main body mask image according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model; and obtaining a main body mask image of the image to be processed according to the output of the main body segmentation model. A training method of the main body segmentation model, an image processing device, a training device of the main body segmentation model, electronic equipment and a readable storage medium are also disclosed, and accuracy of main body segmentation is improved.

Description

Image processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a training method and apparatus for image processing and a main body segmentation model, an electronic device, and a computer readable storage medium.

Background

In recent years, with rapid development of computer technology, image processing technology is diversified, and object detection in acquired images is a common image processing mode, which has wide application in various fields such as image background blurring, image background replacement and the like in smart phones and digital cameras.

In the traditional target detection, the image can be distorted, so that the accuracy of the detection result is reduced.

Disclosure of Invention

The embodiment of the application provides an image processing and main body segmentation model training method and device, electronic equipment and a computer readable storage medium, which are used for processing images in different directions in different rotation modes based on the image direction of an image to be processed and the target processing direction of a main body segmentation model, and carrying out image preprocessing on a training image and a corresponding labeled main body mask image according to the image direction of the training image and the target processing direction of the main body segmentation model in the training process, and improving the accuracy of main body segmentation by associating and matching the preprocessing process of main body segmentation with the training process of the main body segmentation model.

An image processing method, comprising:

Acquiring an image to be processed and an image direction of the image to be processed;

Adjusting the picture direction of the image to be processed according to the matching property of the image direction of the image to be processed and the target processing direction of the main body segmentation model;

inputting the processed image to be processed into the main body segmentation model, wherein the main body segmentation model is a model obtained by training in advance according to a training image and a corresponding labeled main body mask image, and the training process of the main body segmentation model comprises image preprocessing for adjusting the training image and the corresponding labeled main body mask image according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model;

and obtaining a main body mask map of the image to be processed according to the output of the main body segmentation model.

An image processing apparatus comprising:

the acquisition module is used for acquiring the image to be processed and the image direction of the image to be processed;

The direction adjustment module is used for adjusting the picture direction of the image to be processed according to the matching property of the image direction of the image to be processed and the target processing direction of the main body segmentation model;

The mask map module is used for inputting the processed image to be processed into the main body segmentation model, the main body segmentation model is a model obtained by training according to a training image and a corresponding labeled main body mask map in advance, and the training process of the main body segmentation model comprises the image preprocessing of adjusting the training image and the corresponding labeled main body mask map according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model, and the main body mask map of the image to be processed is obtained according to the output of the main body segmentation model.

An electronic device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of:

A computer readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to perform the steps of:

According to the image processing method, the device, the electronic equipment and the computer readable storage medium, the image direction of the image to be processed and the image direction of the image to be processed are obtained, the image direction of the image to be processed is adjusted according to the matching property of the image direction of the image to be processed and the target processing direction of the main body segmentation model, the processed image to be processed is input into the main body segmentation model, the main body segmentation model is a model obtained by training in advance according to the training image and the corresponding labeled main body mask image, the training process of the main body segmentation model comprises the image preprocessing of the training image and the corresponding labeled main body mask image according to the matching property of the image direction of the training image and the target processing direction of the main body segmentation model, the main body mask image is obtained according to the output of the main body segmentation model, the images in different directions are processed by adopting different rotation modes, the training image is also processed according to the matching property of the image direction of the training image and the target processing direction of the main body segmentation model, the problem that the pre-processing of the training image and the corresponding labeled main body mask image is not matched with the target processing direction of the main body segmentation model is solved, and the problem of the error in the process of the label image is avoided, and the error of the image is not being caused by the fact that the label image is not scaled is not matched with the label image, and the size is not matched with the main body segmentation process is generated, and the image is not because the problem is generated.

A training method of a subject segmentation model, comprising:

Acquiring a training sample image, wherein the training sample image comprises a training image and a corresponding labeled label main body mask image;

according to the matching property of the image direction of the training sample image and the target processing direction of the main body segmentation model, the direction of the training sample image is adjusted, and edge filling is carried out according to the proportion of the target edge filling image to obtain an initial processing training sample image;

scaling the initial processing training sample image in equal proportion to obtain a target processing training sample image;

Inputting the target processing training image in the target processing training sample image into a main body segmentation model, outputting a corresponding training main body mask image, and adjusting network parameters of the main body segmentation model according to the difference between the target processing label main body mask image in the target processing training sample image and the training main body mask image so as to obtain a trained main body segmentation model.

A training device for a subject segmentation model, comprising:

the training sample acquisition module is used for acquiring a training sample image, wherein the training sample image comprises a training image and a corresponding marked label main body mask image;

The training direction adjusting module is used for adjusting the direction of the training sample image according to the matching property of the image direction of the training sample image and the target processing direction of the main body segmentation model, and performing edge filling according to the proportion of the target edge filling image to obtain a target processing training sample image;

The training module is used for inputting the target processing training image in the target processing training sample image into a main body segmentation model, outputting a corresponding training main body mask image, and adjusting network parameters of the main body segmentation model according to the difference between the target processing label main body mask image in the target processing training sample image and the training main body mask image so as to obtain a trained main body segmentation model.

According to the matching property of the image direction of the training sample image and the target processing direction of the main body segmentation model, the direction of the training sample image is adjusted, and edge filling is carried out according to the proportion of the target edge filling image to obtain a target processing training sample image;

According to the training method, the device, the electronic equipment and the computer readable storage medium of the main body segmentation model, the training sample images are correspondingly preprocessed in the training process according to the target processing direction of the main body segmentation model and the image direction of the training sample images, different direction adjustment and edge filling are carried out on the training sample images in different directions, the training images and the corresponding marked label main body mask images are simultaneously subjected to consistent adjustment during adjustment, the main body segmentation model obtained through training avoids the problems of edge misalignment, hollowness, false detection and the like of the output main body mask images due to inconsistent image size ratios in different directions, and the accuracy of main body segmentation is improved.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a diagram of an application environment for an image processing method in one embodiment;

FIG. 2 is a flow chart of an image processing method in one embodiment;

FIG. 3 is a schematic view of a transverse forward and reverse image in one embodiment;

FIG. 4 is a schematic view of vertical forward and reverse images in one embodiment;

FIG. 5 is a flow diagram of training of a subject segmentation model in one embodiment;

FIG. 6 is a flow chart of a training method of a subject segmentation model in one embodiment;

FIG. 7a is a schematic diagram of a first subject segmentation model trained to target a portrait shot image in one embodiment;

FIG. 7b is a flow diagram of image processing by a first subject segmentation model in one embodiment;

FIG. 8a is a schematic diagram of a second subject segmentation model trained to target cross-shot image processing direction in one embodiment;

FIG. 8b is a flow diagram of image processing by a second subject segmentation model in one embodiment;

FIG. 9 is a block diagram showing the structure of an image processing apparatus in one embodiment;

FIG. 10 is a block diagram of a training apparatus for a subject segmentation model in one embodiment;

FIG. 11 is a block diagram of the internal structure of an electronic device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

FIG. 1 is a diagram of an application environment for an image processing method in one embodiment. As shown in fig. 1, the application environment includes a terminal 110 and a server 120, the terminal 110 obtains an image to be processed, an image processing request including the image to be processed is sent to the server 120, the server 120 receives the image to be processed and obtains an image direction of the image to be processed, and the image direction of the image to be processed is adjusted according to the matching of the image direction of the image to be processed and a target processing direction of a main body segmentation model; inputting the processed image to be processed into a main body segmentation model, wherein the main body segmentation model is a model obtained by training according to a training image and a corresponding labeled main body mask image in advance, and the training process of the main body segmentation model comprises image preprocessing for adjusting the training image and the corresponding labeled main body mask image according to the matching property of the image direction of the training image and the target processing direction of the main body segmentation model; and obtaining a main body mask map of the image to be processed according to the output of the main body segmentation model, and returning the main body mask map to the terminal 110 by the server 120. The terminal 110 may be a terminal device including a mobile phone, a tablet computer, a PDA (Personal digital assistant), a car-mounted computer, a wearable device, etc. The terminal device can download and apply various types of image resources from the server as images to be processed. Wherein the server 120 may be a server or a cluster of servers.

In some embodiments, the image processing method may be applied to the terminal 110, and the main mask map is generated by directly performing the above steps by the terminal 110.

FIG. 2 is a flow chart of an image processing method in one embodiment. The image processing method shown in fig. 2 may be applied to the terminal 110 or the server 120, and includes:

Step 202, an image to be processed and an image direction of the image to be processed are acquired.

The image to be processed may be an image photographed by the terminal in real time, or may be a preconfigured image, such as an image downloaded in a network, an image in an electronic album, or the like. The image to be processed may be one or more. The type of the image to be processed is not limited, and may be a raw image, a YUV image, an RGB image, a depth image, and the like.

Specifically, the image direction of the image to be processed is determined by the ratio of the width to the height of the image, and is the first direction, i.e., the lateral direction, when the aspect ratio is greater than 1, and is the second direction, i.e., the vertical direction, when the aspect ratio is less than 1. The first direction and the second direction are respectively divided into a forward direction and a reverse direction, the forward shot image accords with the identification direction of human eyes, the reverse shot image is opposite to the identification direction of human eyes, and the first direction and the reverse direction are respectively shown in a schematic diagram of the forward image and the reverse image in fig. 3. A schematic diagram of the forward and reverse images in the second direction is shown in fig. 4, respectively. When the terminal collects images, the image direction set by a user or the images are collected through the default image direction, if the default image direction is transverse, the aspect ratio of the collected images is larger than 1.

In one embodiment, scaling the image to be processed to the target resolution is also included.

Wherein the image to be processed is scaled to a size required for network input of the subject segmentation model, wherein the target resolution is determined by the input requirement of the subject segmentation model.

Specifically, when there are a plurality of subject segmentation models that process images in different directions, one of the corresponding matching subject segmentation models may be selected according to the image direction of the current image to be processed, so that the target resolution is determined according to the input of the matching subject segmentation model. If there are a first subject segmentation model for image processing in a first direction and a second subject segmentation model for processing in a second direction, when the image direction is the first direction, the first subject segmentation model is used as a target subject segmentation model, and a first target resolution is determined according to the target subject segmentation model. And when the image direction is the second direction, taking the second subject segmentation model as a target subject segmentation model, and determining a second target resolution according to the target subject segmentation model. Therefore, a main body segmentation model which is more adaptive to the image to be processed can be selected, and the quality of a main body mask image is improved. It can be understood that even if only one main body segmentation model is provided in the application, since the direction of the image to be processed is updated before the main body segmentation model is input, and the training process of the main body segmentation model includes the image preprocessing of adjusting the training image and the corresponding labeled main body mask image according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model, the main body segmentation model has better compatibility for the images to be processed in different directions and different resolutions.

Step 204, adjusting the image direction of the image to be processed according to the matching between the image direction of the image to be processed and the target processing direction of the main body segmentation model.

And adjusting the picture direction of the image to be processed according to the matching property of the image direction and the target processing direction of the main body segmentation model, and determining the specific angle of adjustment. When determining the specific angle of adjustment, if the image direction includes the forward direction and the reverse direction in the same direction, the image to be processed whose image direction is the same direction also needs to be adjusted in different directions for the forward direction and the reverse direction.

Specifically, the principle of direction adjustment is to enable the adjusted image to be processed to be matched with the target processing direction of the main body segmentation model, so that the influence of different directions on the generation of a main body mask image due to image set is reduced. Because the main body segmentation model is obtained through the training process, different training processes have different image preprocessing on the training image, and when the direction adjustment is carried out, the direction adjustment on the training image in the image preprocessing in the training process is required to be considered, so that the image preprocessing in the actual use process of main body segmentation is matched with the image preprocessing in the training process, and a better main body segmentation effect is achieved. If the target processing direction of the subject segmentation model is horizontal, the image to be processed will be unified into a horizontal size when subject segmentation is performed, and if the target processing direction of the subject segmentation model is vertical, the image to be processed will be unified into a vertical size when subject segmentation is performed.

Step 206, inputting the processed image to be processed into a main body segmentation model, wherein the main body segmentation model is a model obtained by training according to a training image and a corresponding labeled label main body mask image in advance, and the training process of the main body segmentation model comprises image preprocessing for adjusting the training image and the corresponding labeled label main body mask image according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model.

The subject segmentation is to automatically process the region of interest and selectively ignore the region of no interest when facing a scene. The region of interest is referred to as the subject region. The region of interest is typically a foreground region and the region of no interest is typically a background region. The region of interest is typically, but not limited to, a region in which a change in shape, posture, color, or position occurs. The region of interest may be a region in which various objects are located, such as humans, flowers, cats, dogs, cows, blue sky, clouds, and the like. The non-interested region is other regions than the interested region, and can be a region with motion amplitude smaller than a preset threshold value or a region corresponding to a secondary object. Mask refers to a method for fully or partially masking a target image to be processed by using a selected image, graph or object, for example, a background area in the target image can be masked, so as to control the image processing area or processing process. The main mask map may be represented as a binary image or represented by a binary matrix, which is used to distinguish a region where the main object is located from a background region in the image, for example, an element corresponding to the region where the main object is located has a value of 1, and elements in other regions have a value of 0.

Specifically, the training images and the labeled label main mask images are obtained from the training data set, each training image has a corresponding labeled label main mask image, the types of the images are not limited to raw images, YUV images, RGB images, depth images and the like, and the default training image only comprises two directions of a positive direction image in a horizontal shooting direction and a positive direction image in a vertical shooting direction. The training images are relatively random, and in one embodiment, the subjects in the training images are all positive, and there is no inclination of angles, so that the accuracy of the training results is ensured. Training may be performed using training samples of a variety of different aspect ratios.

According to the matching of the image direction of the training image and the target processing direction of the main body segmentation model, the image preprocessing for adjusting the training image and the corresponding labeled label main body mask image refers to comparing the image direction of the training image with the target processing direction of the main body segmentation model, adjusting the direction of the training image and the labeled label main body mask image according to the comparison result, and performing edge filling on the training image after the adjustment direction and the labeled label main body mask image, so that the image after the edge filling accords with a preset image proportion, wherein the preset image proportion can be the configured image proportion or is determined according to the image proportion of the image acquired by the current processing terminal. When the direction is adjusted, the random number and the preset threshold value are quoted, so that the training image and the corresponding marked label main body mask image are randomly adjusted, half of the probability of the overall target of adjustment corresponds to the first rotation direction and the first updating angle, and half of the probability corresponds to the image direction. And the rotation direction in the training process affects the rotation direction of the main body segmentation model for carrying out direction adjustment on the image to be processed according to the image direction in the using process.

In one embodiment, when the image direction of the image to be processed is inconsistent with the target processing direction, when the image direction of the image to be processed is a positive direction, the image to be processed is rotated to a target direction by a target angle, wherein the rotated target direction and target angle are consistent with the rotation direction and target angle of the training image and the corresponding marked label main mask image in the training process.

And step 208, obtaining a main body mask map of the image to be processed according to the output of the main body segmentation model.

Specifically, different image processing can be performed on different areas by applying the main mask image, for example, the interested area in the target image is extracted, the main matting effect is realized, or the different areas are correspondingly processed, the area needing to be processed is determined, and the area which does not participate in the processing is shielded. If the main body is a person, a portrait mask diagram is obtained, and the region outside the portrait region is subjected to blurring post-treatment, so that an effect similar to single-negative shooting is generated. The output main mask image is free from the conditions of edge misalignment, hollowness, false detection and the like through the processing, so that the effect and the precision of the main mask image are improved, the accurate image matting or the blurring effect is improved, and the application effect of the main mask image in various different scenes is improved.

According to the image processing method, the image direction of the image to be processed and the image direction of the image to be processed are obtained, the image direction of the image to be processed is adjusted according to the matching of the image direction of the image to be processed and the target processing direction of the main body segmentation model, the processed image to be processed is input into the main body segmentation model, the main body segmentation model is a model obtained by training according to the training image and the corresponding labeled label main body mask image in advance, the training process of the main body segmentation model comprises the image preprocessing of adjusting the training image and the corresponding labeled label main body mask image according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model, the main body mask image to be processed is obtained according to the output of the main body segmentation model, the images in different directions are processed in different rotation modes, the training image and the corresponding labeled label main body mask image are also preprocessed according to the matching of the image direction of the training image and the target processing direction of the main body segmentation model, and the problem of the error-correcting and the image is avoided due to the fact that the size of the front processing of the main body is matched with the main body segmentation model is not consistent in the image segmentation process, and the problem of the image is solved, and the problem of the error is solved, and the problem is caused by the image is solved.

In one embodiment, after step 202, further comprises: and acquiring a target edge filling image proportion in the image preprocessing of the main body segmentation model, and scaling the image to be processed to an image resolution matching the target edge filling image proportion.

Specifically, the target edge filling image proportion in the image preprocessing of the main body segmentation model is generally an image proportion with stronger compatibility or an image proportion with high use frequency, and the image to be processed is scaled according to the target edge filling image proportion, so that the resolution of the image obtained after scaling meets the input requirement of the main body segmentation model. When the target processing direction of the subject segmentation model is the lateral direction, the edge-fill image scale may be the image scale of the lateral direction, such as 4/3, 16/9, or other image scale with an aspect ratio greater than 1. When the target processing direction of the subject segmentation model is the portrait direction, the edge-fill image scale may be the image scale of the portrait direction, such as 3/4, 9/16, or other image scale with an aspect ratio less than 1. And when the resolution of the image required by the input of the main body segmentation model is smaller than that of the image to be processed, performing reduction processing on the image to be processed. When the resolution of the image required by the input of the main body segmentation model is larger than that of the image to be processed, the image to be processed is amplified.

In this embodiment, by the target edge filling image proportion in the image preprocessing of the subject segmentation model, the image to be processed can be quickly scaled to the image resolution matching the target edge filling image proportion, and the training process of the subject segmentation model is associated with the preprocessing in the use process of the subject segmentation model.

In one embodiment, the training sample image includes a training image and a corresponding labeled label subject mask map, as shown in fig. 5, and the training of the subject segmentation model includes the steps of:

Step 302, determining whether the training sample image is the target processing direction of the subject segmentation model.

Specifically, the training sample image comprises a training image and a corresponding labeled label main body mask image, wherein the training image and the labeled label main body mask image have a matching relationship, the image directions of the training image and the labeled label main body mask image are consistent, the subsequent pretreatment process is also consistent, and the labeled label main body mask image is a real main body mask image. The target processing direction of the subject segmentation model is generally an image direction with high compatibility or an image direction with high frequency of use. Wherein the target process direction may be a lateral direction or a vertical direction.

Step 304, when the image direction of the training sample image is the target processing direction, a first random number is obtained, a first direction updating angle of the training sample image is determined according to a comparison result of the first random number and a preset threshold value, the training sample image is subjected to direction updating according to the first direction updating angle, and the training sample image after the direction updating is subjected to edge filling according to the target edge filling image proportion to obtain an initial processing training sample image.

Specifically, the first random data may be generated by uniformly distributing random numbers or gaussian random numbers, the preset threshold may be defined by a range of random numbers, and the comparison result is two mutually exclusive results, where the first random number is greater than the preset threshold and the first random number is less than or equal to the preset threshold. The principle is defined that the probability of the occurrence of the two comparison results is similar, namely the number of the first results is approximately equal to the number of the second results, and the probability of the subsequent updating branches entering different directions is ensured to be similar. And when the comparison result is the first result, keeping the direction of the image unchanged. And when the comparison result is the second result, rotating the training sample image by a preset angle, wherein the rotating direction can be customized and can be clockwise or anticlockwise. And carrying out edge filling on the training sample image with updated direction according to the proportion of the target edge filling image to obtain an initial processing training sample image, wherein the long edge can be kept unchanged during filling, the short edge is filled, if the training sample image is an RGB image, three channels are respectively filled, the filling value of each channel is 127.5, and the filling value is the average value of the three channels of the training image. The filling value of the marked label main mask graph is an neglected class value, wherein the neglected class value refers to a value of main body recognition result of the main body mask graph, and if the main body mask graph comprises 0 and 1, 255 is used for filling the main body mask graph.

And 306, when the image direction of the training sample image is not the target processing direction, acquiring a second random number, determining a second direction updating angle of the training sample image according to a comparison result of the second random number and a preset threshold value, carrying out direction updating on the training sample image according to the second direction updating angle, and carrying out edge filling on the training sample image after the direction updating according to the target edge filling image proportion to obtain an initial processing training sample image.

Specifically, the second random data may be generated by uniformly distributing random numbers or gaussian random numbers, the preset threshold may be defined by a range of random numbers, and the comparison result is two mutually exclusive results, where the two results are respectively that the second random number is greater than the preset threshold and the second random number is less than or equal to the preset threshold. The principle is defined that the probability of the occurrence of the two comparison results is similar, namely the number of the third results is approximately equal to the number of the fourth results, and the probability of the subsequent updating branches entering different directions is ensured to be similar. And when the comparison result is a third result, keeping the direction of the image unchanged. And when the comparison result is a fourth result, rotating the training sample image by a preset angle, wherein the rotating direction can be customized and can be clockwise or anticlockwise. Wherein the predetermined angle and the direction of rotation are consistent with the predetermined angle and the direction of rotation in step 304. And performing edge filling on the training sample image with the updated direction according to the target edge filling image proportion to obtain an initial processing training sample image, wherein the filling mode can refer to the filling mode in the step 304.

Step 308, scaling the initial processed training sample image in equal proportion to obtain a target processed training sample image, and inputting the target processed training sample image into the subject segmentation model for training to obtain a trained subject segmentation model.

Specifically, since the previous step has edge-filled the image in the target edge-filled image scale, the aspect ratio need not be changed during scaling, and only an equal scale scaling is required. The initial processing training sample image is scaled to the size required by the main body segmentation model network, if the input width and height of the first main body segmentation model network are 480 x 640, the initial processing training sample image is required to be scaled to 480 x 640. The input width of the second body segmentation model network is 640×480, and the initial processing training sample image needs to be scaled to 640×480. The algorithm employed by the subject segmentation model may be, but is not limited to, deeplab series segmentation algorithms, U-Net, FCN (Fully Convolutional Networks), and the like. The main body segmentation model comprises Encoder feature coding modules and a Decoder target template generating module, training data of the model is composed of a main body dataset, a main body is a category, and a background is a category. The method comprises the steps of punishing by adopting a cost function such as softmax cross entropy, inputting a target processing training image in a target processing training sample image into a main body segmentation model, outputting a corresponding training main body mask image, substituting the target processing label main body mask image and the training main body mask image in the target processing training sample image into the cost function to calculate a loss value, reversely propagating and updating convolutional neural network parameters of the main body segmentation model according to the loss value, iterating continuously until the network meets convergence conditions, and finishing the iteration to obtain the trained main body segmentation model.

In this embodiment, according to the target processing direction of the main body segmentation model and the image direction of the training sample image, corresponding preprocessing is performed on the training sample image in the training process, different direction adjustment and edge filling are performed on the training sample image in different directions, and when adjustment is performed, consistent adjustment is performed on the training image and the corresponding labeled main body mask image, so that the main body segmentation accuracy of the main body segmentation model obtained by training is high.

In one embodiment, determining the first direction update angle of the training sample image according to the comparison result of the first random number and the preset threshold in step 304 includes: when the first random number is larger than a preset threshold value, the first direction updating angle is 0 degree; otherwise, the first direction update angle is rotated 90 degrees.

Specifically, when the image direction of the training sample image is the target processing direction, the direction of the training sample image is adjusted by comparing the first random number with a preset threshold. Random numbers between 0 and 1 can be generated, in theory, shooting probabilities of the horizontal shooting and the vertical shooting of the scene are equal, a preset threshold value can be set to be 0.5, and when the first random number is larger than 0.5, the first direction updating angle is 0 degrees, so that the direction of the training sample image is kept unchanged. When the first random number is less than or equal to 0.5, the first direction updating angle is rotated 90 degrees, and can be rotated 90 degrees clockwise or rotated 90 degrees counterclockwise. The horizontal shot image can be converted into the vertical shot image or the vertical shot image can be converted into the horizontal shot image by rotating by 90 degrees.

In this embodiment, when the image direction of the training sample image is the target processing direction, the training sample image is updated according to a preset rule by introducing a random number, so as to improve the detection compatibility of the main body segmentation model on images in different directions.

In one embodiment, determining the second direction update angle of the training sample image according to the comparison result of the second random number and the preset threshold in step 306 includes: when the second random number is larger than the preset threshold value, the second direction updating angle is 0 degree, otherwise, the second direction updating angle is rotated by 90 degrees.

Specifically, when the image direction of the training sample image is not the target processing direction, the direction of the training sample image is adjusted by comparing the second random number with a preset threshold. Random numbers between 0 and 1 can be generated, in theory, shooting probabilities of the horizontal shooting and the vertical shooting of the scene are equal, a preset threshold value can be set to be 0.5, and when the second random number is larger than 0.5, the second direction updating angle is 0 degrees, so that the direction of the training sample image is kept unchanged. When the second random number is less than or equal to 0.5, the second direction updating angle is rotated 90 degrees, and can be rotated 90 degrees clockwise or rotated 90 degrees counterclockwise. And the direction of rotation remains the same as in step 304. The horizontal shot image can be converted into the vertical shot image or the vertical shot image can be converted into the horizontal shot image by rotating by 90 degrees.

In this embodiment, when the image direction of the training sample image is not the target processing direction, the training sample image is updated according to a preset rule by introducing a random number, so as to improve the detection compatibility of the main body segmentation model on images in different directions.

In one example, step 204 includes: and acquiring a target processing direction of the main body segmentation model, when the image direction of the image to be processed is consistent with the target processing direction, keeping the direction of the image to be processed unchanged when the image direction of the image to be processed is a positive direction, and rotating the image to be processed by 180 degrees when the image direction of the image to be processed is a reverse direction. When the image direction of the image to be processed is inconsistent with the target processing direction, rotating the image to be processed to a third direction by 90 degrees when the image direction of the image to be processed is a positive direction, and rotating the image to be processed to the reverse direction of the third direction by 90 degrees when the image direction of the image to be processed is a reverse direction, wherein the third direction is consistent with the rotating direction in the image pre-processing in the training process of the main body segmentation model.

Specifically, when the image direction of the image to be processed is consistent with the target processing direction, when the image direction of the image to be processed is a positive direction, it is indicated that the image to be processed does not need to be processed, and the direction of the image to be processed can be directly kept unchanged. When the image direction of the image to be processed is the reverse direction, the image to be processed is rotated 180 degrees to adjust the reverse direction to the forward direction. When rotated, the rotation may be 180 degrees clockwise or 180 degrees counterclockwise. When the image direction of the image to be processed is inconsistent with the target processing direction, the image to be processed is rotated 90 degrees to the third direction when the image direction of the image to be processed is the positive direction, and is rotated 90 degrees to the opposite direction of the third direction when the image direction of the image to be processed is the opposite direction. The third direction is consistent with the rotation direction in the image pre-processing during the training of the subject segmentation model, that is, if the first direction updating angle in step 304 is clockwise, the third direction is clockwise, and if the first direction updating angle in step 304 is counterclockwise, the third direction is counterclockwise. By rotating by 90 degrees, the image to be processed which does not coincide with the target processing direction can be adjusted to coincide with the target processing direction.

In this embodiment, according to the image direction of the image to be processed, the target processing direction, and the forward and backward direction of the image to be processed, the direction of the image to be processed is correspondingly adjusted, so that the input image accords with the target processing direction of the main body segmentation model, so as to improve the quality of the main body mask image.

In one embodiment, the determination of the target process direction includes the steps of: and acquiring a default image storage direction of the current equipment, and taking the default image storage direction as a target processing direction.

Specifically, the device defaults to store the transverse beats in the form of transverse beats regardless of the direction of image acquisition, and the target processing direction of the main body segmentation model during training is transverse beats. The device defaults to store the vertical shooting, no matter what direction is during image acquisition, the device stores the vertical shooting, and the target processing direction of the main body segmentation model is the vertical shooting during training.

In this embodiment, the default image storage direction is used as the target processing direction, so that the body segmentation model is suitable for the image direction of the device, the number of times of direction update is reduced, and the efficiency is improved.

In one embodiment, the determination of the target edge fill image scale includes the steps of: and acquiring the image acquisition proportion of the current equipment, and taking the image acquisition proportion as the target edge filling image proportion.

Specifically, the image acquisition proportion is taken as the target edge filling image proportion, the original image acquired by equipment accords with the target edge filling image proportion, edge filling is not needed, scaling is only needed subsequently, scaling can be performed in equal proportion during subsequent scaling, the image proportion of the original image is not needed to be changed, and distortion caused by image proportion change on the image is avoided.

In this embodiment, the target edge filling image proportion is determined according to the image acquisition proportion of the current device, so that distortion caused by image proportion change on the image is reduced, and quality of a main mask image obtained by a main segmentation model is improved.

In one embodiment, after step 202, further comprises: and carrying out normalization processing on the image to be processed.

Specifically, the normalization algorithm can be customized, and the three channels of the image RGB can be normalized by means of mean-reduction and variance division, for example, for the r value of the image, the normalized value is obtained by (r-127.5)/127.5, and for the g value and the b value, the normalization value corresponding to each pixel point is obtained by the same operation. 127.5 may be derived from the mean and variance of all channels of all pictures in the preset gallery, so that the preset gallery may be different, and the mean and variance may be different.

In this embodiment, the quality of the main mask image output by the main segmentation model can be higher by performing normalization processing on the image to be processed.

In one embodiment, inputting the target processing training sample image into the subject segmentation model for training to obtain a trained subject segmentation model comprises: when the target processing training sample image is a training image, the training image is normalized.

Specifically, the normalization algorithm can be customized, and the three channels of the image RGB can be normalized by means of mean-reduction and variance division, for example, for the r value of the image, the normalized value is obtained by (r-127.5)/127.5, and for the g value and the b value, the normalization value corresponding to each pixel point is obtained by the same operation. Where 127.5 is the mean and variance of the individual channels from all pictures of the training set, the training set is different, and the mean and variance may also be different. The normalization process when training is performed can be consistent with the normalization process performed on the input image before the body segmentation model is actually used later.

In this embodiment, the training image is normalized, so that the CNN network can be effectively converged, and the network overfitting is prevented.

In one embodiment, a training method of a body segmentation model is provided, so as to be applied to the terminal 110 or the server 120, as shown in fig. 6, including:

Step 402, a training sample image is acquired, the training sample image including a training image and a corresponding labeled label body mask map.

And step 404, adjusting the direction of the training sample image according to the matching of the image direction of the training sample image and the target processing direction of the main body segmentation model, and performing edge filling according to the target edge filling image proportion to obtain a target processing training sample image.

In one embodiment, further comprising: the target processing training sample image is scaled equally.

Step 406, inputting the target processing training image in the target processing training sample image into a main body segmentation model, outputting a corresponding training main body mask image, and adjusting network parameters of the main body segmentation model according to the difference between the target processing label main body mask image and the training main body mask image in the target processing training sample image to obtain a trained main body detection model.

In this embodiment, according to the target processing direction of the main body segmentation model and the image direction of the training sample image, corresponding preprocessing is performed on the training sample image in the training process, different direction adjustment and edge filling are performed on the training sample image in different directions, during adjustment, the training image and the corresponding labeled main body mask image are simultaneously adjusted in a consistent manner, the main body segmentation model obtained through training avoids the problems of edge misalignment, hollowness, false detection and the like of the main body mask image output due to inconsistent image size ratios acquired in different directions, and the accuracy of main body segmentation is improved.

In a specific embodiment, an image processing method is provided, where a first subject segmentation model with a vertical direction as a target processing direction is obtained by training a training method of a subject segmentation model as shown in fig. 7a, and the first image processing method shown in fig. 7b uses the first subject segmentation model obtained by training in fig. 7a to process an image, so as to obtain a subject mask map. The specific training process shown in fig. 7a is as follows:

1. and acquiring a training sample image from the training data set, wherein the training sample image comprises a training image and a corresponding labeled label main body mask image, and the default training image only comprises a transverse shooting forward direction and a vertical shooting forward direction.

2. Judging whether the current training sample image is a vertically shot image or not, if so, generating a first random number, and when the first random number is larger than 0.5, directly performing edge filling to a target edge filling image proportion, such as an image proportion with the aspect ratio of 4/3. And when the first random number is smaller than or equal to 0.5, rotating the current training sample image by 90 degrees, and performing edge filling to the target edge filling image proportion. The rotation can be clockwise or anticlockwise. The initial processing training sample image is obtained through the processing.

3. If the current training sample image is not the vertically shot image, generating a second random number, and directly performing edge filling to the target edge filling image proportion when the second random number is larger than 0.5. And when the second random number is smaller than or equal to 0.5, rotating the current training sample image by 90 degrees, and performing edge filling to the target edge filling image proportion. The initial processing training sample image is obtained through the processing.

4. And scaling the initial processing training sample image in an equal proportion to obtain a target processing training sample image, and normalizing the target processing training image in the target processing training sample image.

5. Inputting the normalized target processing training image into a main body segmentation model to obtain an output training main body mask image, calculating a cost value according to the training main body mask image and a target processing label main body mask image in a target processing training sample image, and adjusting a main body detection model according to back propagation of the cost value until convergence conditions are met to obtain a trained first main body segmentation model taking the vertical direction as the target processing direction.

The first image processing method shown in fig. 7b processes the image by using the first subject segmentation model generated in the above steps, to obtain a subject mask map. The specific process shown in fig. 7b is as follows:

1. Obtaining an image to be processed and a photographing direction, and scaling the processed image to a target resolution which is consistent with the input of a main body segmentation model to obtain a scaled image;

2. The scaled image is normalized.

3. Judging the image direction through the photographing direction, if the image to be processed is in the forward direction of the transverse photographing direction, rotating 90 degrees towards the third direction, if the image to be processed is rotated 90 degrees clockwise in the step 2 of the training process, the image is rotated 90 degrees clockwise in the step, and if the image to be processed is rotated 90 degrees anticlockwise in the step 2 of the training process, the image is rotated 90 degrees anticlockwise in the step. If the direction of the cross beat is reversed, the rotation is 90 degrees in the reverse direction of the third direction, if the rotation is 90 degrees clockwise in step 2 of the training process, the rotation is 90 degrees counterclockwise in this step, and if the rotation is 90 degrees counterclockwise in step 2 of the training process, the rotation is 90 degrees clockwise in this step.

4. If the image to be processed is in the vertical shooting forward direction, the direction is kept unchanged, and if the image to be processed is in the vertical shooting reverse direction, the image to be processed is rotated 180 degrees clockwise or anticlockwise.

5. And inputting the processed scaled image into a first main body segmentation model to obtain an output main body mask map.

In this embodiment, a first main body segmentation model with a vertical direction as a target processing direction is obtained through training, different direction adjustment and edge filling are performed on training sample images in different directions in the training process, the training images and corresponding labeled main body mask images are simultaneously subjected to consistent adjustment during adjustment, the first main body segmentation model is obtained through training, a preprocessing process matched with the first main body segmentation model is performed on an image to be processed during actual main body segmentation, the image to be processed is converted into an image matched with the target processing direction of the first main body segmentation model, and then the image to be processed is input into the first main body segmentation model, and the preprocessing process of main body segmentation is matched with the training process of the main body segmentation model, so that problems of edge misalignment, hollowness, false detection and the like of the output main body mask images are avoided due to inconsistent image size proportions acquired in different directions, and the accuracy of main body segmentation is improved.

In a specific embodiment, an image processing method is provided, where a second subject segmentation model with a transverse direction as a target processing direction is obtained by training a training method of the subject segmentation model as shown in fig. 8a, and the second image processing method as shown in fig. 8b uses the second subject segmentation model obtained by training in fig. 8a to process an image, so as to obtain a subject mask map. The specific training process shown in fig. 8a is as follows:

2. Judging whether the current training sample image is a transverse image, if so, generating a first random number, and directly performing edge filling to a target edge filling image proportion, such as an image proportion with the aspect ratio of 4/3, when the first random number is larger than 0.5. And when the first random number is smaller than or equal to 0.5, rotating the current training sample image by 90 degrees, and performing edge filling to the target edge filling image proportion. The rotation can be clockwise or anticlockwise. The initial processing training sample image is obtained through the processing.

3. If the current training sample image is not a cross shot image, generating a second random number, and directly performing edge filling to the target edge filling image proportion when the second random number is larger than 0.5. And when the second random number is smaller than or equal to 0.5, rotating the current training sample image by 90 degrees, and performing edge filling to the target edge filling image proportion. The initial processing training sample image is obtained through the processing.

5. Inputting the normalized target processing training image into a main body segmentation model to obtain an output training main body mask image, calculating a cost value according to the training main body mask image and a target processing label main body mask image in a target processing training sample image, and adjusting a main body detection model according to back propagation of the cost value until convergence conditions are met to obtain a trained second main body segmentation model taking the transverse direction as the target processing direction.

The second image processing method shown in fig. 8b processes the image by using the second subject segmentation model generated in the above step, to obtain a subject mask map. The specific process shown in fig. 8b is as follows:

2. The scaled image is normalized.

3. Judging the image direction through the photographing direction, if the image to be processed is in the forward direction of the vertical photographing direction, rotating 90 degrees towards the third direction, if the image to be processed is rotated 90 degrees clockwise in the step 2 of the training process, the image is rotated 90 degrees clockwise in the step, and if the image to be processed is rotated 90 degrees anticlockwise in the step 2 of the training process, the image is rotated 90 degrees anticlockwise in the step. If the vertical shooting direction is reversed, the rotation is 90 degrees in the reverse direction of the third direction, if the rotation is 90 degrees clockwise in the step 2 of the training process, the rotation is 90 degrees counterclockwise in the step, and if the rotation is 90 degrees counterclockwise in the step 2 of the training process, the rotation is 90 degrees clockwise in the step.

4. If the image to be processed is in the horizontal forward direction, the direction is kept unchanged, and if the image to be processed is in the horizontal reverse direction, the image to be processed is rotated 180 degrees clockwise or anticlockwise.

5. And inputting the processed scaled image into a second main body segmentation model to obtain an output main body mask map.

In this embodiment, a second main body segmentation model with a transverse direction as a target processing direction is obtained through training, different direction adjustment and edge filling are performed on training sample images in different directions in the training process, the training images and corresponding labeled main body mask images are simultaneously subjected to consistent adjustment during adjustment, the second main body segmentation model is obtained through training, a preprocessing process matched with the second main body segmentation model is performed on an image to be processed during actual main body segmentation, the image to be processed is converted into an image matched with the target processing direction of the second main body segmentation model, and then the image to be processed is input into the second main body segmentation model, and the preprocessing process of main body segmentation is matched with the training process of the main body segmentation model, so that problems of edge misalignment, hollowness, false detection and the like of the output main body mask images are avoided due to inconsistent image size proportions acquired in different directions, and the accuracy of main body segmentation is improved.

It should be understood that, although the steps in the flowcharts of fig. 2, 5, and 6 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2, 5, 6 may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed sequentially, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

Fig. 9 is a block diagram of the structure of an image processing apparatus 900 of an embodiment. As shown in fig. 9, an image processing apparatus 900 includes: an acquisition module 902, a direction adjustment module 904, and a mask map module 906. Wherein:

an acquiring module 902, configured to acquire an image to be processed and an image direction of the image to be processed.

The direction adjustment module 904 is configured to adjust a picture direction of the image to be processed according to matching between the image direction of the image to be processed and a target processing direction of the subject segmentation model.

The mask map module 906 is configured to input the processed image to the main body segmentation model, where the main body segmentation model is a model obtained by training in advance according to a training image and a corresponding labeled main body mask map, and the training process of the main body segmentation model includes performing image preprocessing for adjusting the training image and the corresponding labeled main body mask map according to the matching between the image direction of the training image and the target processing direction of the main body segmentation model, and obtaining the main body mask map of the image to be processed according to the output of the main body segmentation model.

The image processing device 900 in this embodiment adjusts the image direction of the image to be processed according to the matching between the image direction of the image to be processed and the target processing direction of the main body segmentation model by acquiring the image direction of the image to be processed and the image direction of the image to be processed, inputs the processed image to be processed into the main body segmentation model, wherein the main body segmentation model is a model obtained by training in advance according to the matching between the training image and the corresponding labeled main body mask image, the training process of the main body segmentation model comprises the image preprocessing of adjusting the training image and the corresponding labeled main body mask image according to the matching between the image direction of the training image and the target processing direction of the main body segmentation model, obtains the main body mask image of the image to be processed according to the output of the main body segmentation model, processes the images in different acquisition directions by adopting different rotation modes, and also carries out the image preprocessing on the training image and the corresponding labeled main body mask image according to the matching between the image direction of the training image and the target processing direction of the main body segmentation model, and the problem of the error-correction caused by the error-correction of the image segmentation model is avoided, and the error-correction of the image is avoided due to the error-correction of the image, the image segmentation process.

In one embodiment, the apparatus further comprises a scaling module for obtaining a target edge-filled image scale in the image pre-processing of the subject segmentation model, scaling the image to be processed to an image resolution matching the target edge-filled image scale.

The image processing apparatus 900 in this embodiment can quickly scale the image to be processed to an image resolution matching the target edge filling image ratio by the target edge filling image ratio in the image preprocessing of the subject segmentation model, and correlate the training process of the subject segmentation model with the preprocessing in the use process of the subject segmentation model.

In one embodiment, the training sample image includes the training image and a corresponding labeled label body mask map, and the apparatus further includes: the training module is used for judging whether the training sample image is the target processing direction of the main body segmentation model or not; when the image direction of the training sample image is the target processing direction, a first random number is obtained, a first direction updating angle of the training sample image is determined according to a comparison result of the first random number and a preset threshold value, the training sample image is subjected to direction updating according to the first direction updating angle, and the training sample image subjected to direction updating is subjected to edge filling according to the target edge filling image proportion to obtain an initial processing training sample image; when the image direction of the training sample image is not the target processing direction, acquiring a second random number, determining a second direction updating angle of the training sample image according to a comparison result of the second random number and a preset threshold value, carrying out direction updating on the training sample image according to the second direction updating angle, and carrying out edge filling on the training sample image after the direction updating according to the target edge filling image proportion to obtain an initial processing training sample image; scaling the initial processing training sample image in equal proportion to obtain a target processing training sample image; inputting the target processing training sample image into a main body segmentation model for training to obtain a trained main body segmentation model.

The image processing apparatus 900 in this embodiment performs corresponding preprocessing on the training sample image during training according to the target processing direction of the main body segmentation model and the image direction of the training sample image, performs different direction adjustment and edge filling on the training sample image in different directions, and performs consistent adjustment on the training image and the corresponding labeled main body mask image at the same time during adjustment, so that the main body segmentation accuracy of the main body segmentation model obtained by training is high.

In one embodiment, the training module is further configured to update the first direction by 0 degrees when the first random number is greater than a preset threshold, and otherwise rotate by 90 degrees.

In one embodiment, the training module is further configured to update the second direction by 0 degrees when the second random number is greater than a preset threshold, and otherwise rotate by 90 degrees.

In the image processing apparatus 900 in this embodiment, when the image direction of the training sample image is not the target processing direction, the training sample image is updated according to a preset rule by introducing a random number, so as to improve the detection compatibility of the main body segmentation model on images in different directions.

In one embodiment, the direction adjustment module 904 is further configured to obtain the target processing direction of the subject segmentation model, when the image direction of the image to be processed is consistent with the target processing direction, keep the direction of the image to be processed unchanged when the image direction of the image to be processed is a positive direction, rotate the image to be processed by 180 degrees when the image direction of the image to be processed is a negative direction, rotate the image to be processed by 90 degrees when the image direction of the image to be processed is not consistent with the target processing direction, rotate the image to be processed by 90 degrees in a third direction when the image direction of the image to be processed is a positive direction, and rotate the image to be processed by 90 degrees in a negative direction of the third direction when the image direction of the image to be processed is a negative direction, wherein the third direction is consistent with the rotation direction in the image pre-processing in the training process of the subject segmentation model.

In one embodiment, the target processing direction is determined by acquiring a default image storage direction of the current device, and using the default image storage direction as the target processing direction.

In one embodiment, the target edge-filled image scale is determined by acquiring an image acquisition scale of the current device, and taking the image acquisition scale as the target edge-filled image scale.

In one embodiment, the apparatus further comprises:

and the normalization module is used for carrying out normalization processing on the image to be processed.

In one embodiment, the training module is further configured to normalize the training image when the target processing training sample image is a training image.

Fig. 10 is a block diagram showing a structure of a training apparatus 1000 for a body segmentation model according to an embodiment. As shown in fig. 10, a training apparatus 1000 for a subject segmentation model includes: a training sample acquisition module 1002, a training direction adjustment module 1004, and a training module 1006. Wherein:

The training sample obtaining module 1002 is configured to obtain a training sample image, where the training sample image includes a training image and a corresponding labeled label main mask map.

The training direction adjustment module 1004 is configured to adjust a direction of the training sample image according to a matching property between an image direction of the training sample image and a target processing direction of the main body segmentation model, and perform edge filling according to a target edge filling image proportion to obtain a target processing training sample image.

The training module 1006 is configured to input a target processing training image in the target processing training sample image into the main body segmentation model, output a corresponding training main body mask map, and adjust network parameters of the main body segmentation model according to a difference between the target processing label main body mask map and the training main body mask map in the target processing training sample image, so as to obtain a trained main body segmentation model.

According to the training device 1000 of the main body segmentation model in the embodiment, according to the target processing direction of the main body segmentation model and the image direction of the training sample image, corresponding preprocessing is performed on the training sample image in the training process, different direction adjustment and edge filling are performed on the training sample image in different directions, and when the adjustment is performed, the training image and the corresponding labeled main body mask image are simultaneously subjected to consistent adjustment, the main body segmentation model obtained through training avoids the problems of edge misalignment, hollowness, false detection and the like of the main body mask image output due to the fact that the size proportion of the acquired images in different directions is inconsistent, and the accuracy of main body segmentation is improved.

In one embodiment, the apparatus further comprises: and the training scaling module is used for scaling the target processing training sample image in an equal proportion.

In this embodiment, the target processing training sample image is scaled in equal proportion, and then the main body segmentation model is input, so that the image is conveniently adjusted to be uniform in size, and the processing efficiency is improved.

For specific limitations of the image processing apparatus and the training apparatus for the subject segmentation model, reference may be made to the above limitations of the image processing method and the training method for the subject segmentation model, and details thereof are not repeated here. The respective modules in the image processing apparatus and the training apparatus of the subject segmentation model may be realized in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

Fig. 11 is a schematic diagram of an internal structure of an electronic device in one embodiment. As shown in fig. 11, the electronic device includes a processor and a memory connected via a system bus. Wherein the processor is configured to provide computing and control capabilities to support operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program is executable by a processor for implementing the image processing method or the training method of the subject segmentation model provided in the above respective embodiments. The internal memory provides a cached operating environment for operating system computer programs in the non-volatile storage medium. The electronic device may be a cell phone, a server, etc.

The implementation of each module in the image processing device and the training device of the main body segmentation model provided in the embodiment of the application can be in the form of a computer program. The computer program may run on a terminal or a server. Program modules of the computer program may be stored in the memory of the terminal or server. Which when executed by a processor, performs the steps of the method described in the embodiments of the application.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform an image processing method or a training method of a subject segmentation model.

A computer program product comprising instructions which, when run on a computer, cause the computer to perform an image processing method or a training method of a subject segmentation model.

Any reference to memory, storage, database, or other medium used in the present application may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1. An image processing method, comprising:

Adjusting the picture direction of the image to be processed according to the matching property of the image direction of the image to be processed and the target processing direction of the main body segmentation model, so that the adjusted image to be processed is matched with the target processing direction of the main body segmentation model, inputting the processed image to be processed into the main body segmentation model, wherein the main body segmentation model is a model which is obtained by training according to a training image and a corresponding labeled main body mask image in advance, and the training process of the main body segmentation model comprises image preprocessing for adjusting the training image and the corresponding labeled main body mask image according to the matching property of the image direction of the training image and the target processing direction of the main body segmentation model;

2. The method according to claim 1, further comprising, after the acquiring the image to be processed and the image direction of the image to be processed:

acquiring a target edge filling image proportion in the image preprocessing of the main body segmentation model;

and scaling the image to be processed to an image resolution which is matched with the target edge filling image in proportion.

3. The method of claim 1, wherein training sample images comprise the training images and corresponding labeled label subject mask maps, and training of the subject segmentation model comprises the steps of:

Judging whether the training sample image is the target processing direction of the main body segmentation model or not;

when the image direction of the training sample image is the target processing direction, a first random number is obtained, a first direction updating angle of the training sample image is determined according to a comparison result of the first random number and a preset threshold value, the training sample image is subjected to direction updating according to the first direction updating angle, and the training sample image subjected to direction updating is subjected to edge filling according to the target edge filling image proportion to obtain an initial processing training sample image;

When the image direction of the training sample image is not the target processing direction, a second random number is obtained, a second direction updating angle of the training sample image is determined according to a comparison result of the second random number and a preset threshold value, the training sample image is subjected to direction updating according to the second direction updating angle, and the training sample image subjected to direction updating is subjected to edge filling according to the target edge filling image proportion to obtain an initial processing training sample image;

and inputting the target processing training sample image into a main body segmentation model for training to obtain a trained main body segmentation model.

4. A method according to claim 3, wherein determining the first direction update angle of the training sample image according to the comparison result of the first random number and a preset threshold value comprises:

When the first random number is larger than the preset threshold value, the first direction updating angle is 0 degree;

Otherwise, the first direction updating angle is rotated by 90 degrees.

5. A method according to claim 3, wherein determining the second direction update angle of the training sample image according to the comparison result of the second random number and a preset threshold value comprises:

When the second random number is larger than the preset threshold value, the second direction updating angle is 0 degree;

Otherwise, the second direction updating angle is rotated by 90 degrees.

6. A method according to claim 3, wherein said adjusting the picture direction of the image to be processed according to the matching of the image direction of the image to be processed with the target processing direction of the subject segmentation model comprises:

acquiring the target processing direction of a main body segmentation model;

When the image direction of the image to be processed is consistent with the target processing direction, the direction of the image to be processed is kept unchanged when the image direction of the image to be processed is a positive direction, and the image to be processed is rotated 180 degrees when the image direction of the image to be processed is a negative direction;

When the image direction of the image to be processed is inconsistent with the target processing direction, rotating the image to be processed to a third direction by 90 degrees when the image direction of the image to be processed is a positive direction, and rotating the image to be processed to the reverse direction of the third direction by 90 degrees when the image direction of the image to be processed is a reverse direction, wherein the third direction is consistent with the rotation direction in the image pre-processing in the training process of the main body segmentation model.

7. The method according to any one of claims 1 to 6, wherein the determination of the target process direction comprises the steps of:

and acquiring a default image storage direction of the current equipment, and taking the default image storage direction as the target processing direction.

8. The method according to any one of claims 2 to 6, wherein the determination of the target edge-filled image scale comprises the steps of:

And acquiring the image acquisition proportion of the current equipment, and taking the image acquisition proportion as the target edge filling image proportion.

9. The method according to any one of claims 1 to 6, further comprising, after the acquiring the image to be processed and the image direction of the image to be processed:

and carrying out normalization processing on the image to be processed.

10. The method of any one of claims 3 to 6, wherein inputting the target processing training sample image into a subject segmentation model for training to obtain a trained subject segmentation model comprises;

and when the target processing training sample image is a training image, carrying out normalization processing on the training image.

11. A method of training a subject segmentation model, comprising:

12. An image processing apparatus, comprising:

The direction adjustment module is used for adjusting the picture direction of the image to be processed according to the matching property of the image direction of the image to be processed and the target processing direction of the main body segmentation model, so that the adjusted image to be processed is matched with the target processing direction of the main body segmentation model;

13. A training device for a subject segmentation model, comprising:

The training direction adjusting module is used for adjusting the direction of the training sample image according to the matching property of the image direction of the training sample image and the target processing direction of the main body segmentation model, and performing edge filling according to the target edge filling image proportion to obtain an initial processing training sample image;

the training scaling module is used for scaling the initial processing training sample image in an equal proportion to obtain a target processing training sample image;

14. An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of the method of any of claims 1 to 11.

15. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

16. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 11.