CN109961403B - Photo adjusting method and device, storage medium and electronic equipment - Google Patents

Photo adjusting method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN109961403B
CN109961403B CN201711407286.5A CN201711407286A CN109961403B CN 109961403 B CN109961403 B CN 109961403B CN 201711407286 A CN201711407286 A CN 201711407286A CN 109961403 B CN109961403 B CN 109961403B
Authority
CN
China
Prior art keywords
photo
adjusted
local object
preset
adjusting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201711407286.5A
Other languages
Chinese (zh)
Other versions
CN109961403A (en
Inventor
陈岩
刘耀勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to CN201711407286.5A priority Critical patent/CN109961403B/en
Priority to PCT/CN2018/116746 priority patent/WO2019120025A1/en
Publication of CN109961403A publication Critical patent/CN109961403A/en
Application granted granted Critical
Publication of CN109961403B publication Critical patent/CN109961403B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)

Abstract

The application discloses a photo adjusting method and device, a storage medium and electronic equipment. The photo adjusting method comprises the following steps: acquiring a photo to be adjusted; performing semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model to obtain a photo segmentation result; determining the local object category contained in the photo to be adjusted according to the photo segmentation result; and acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type by using each target adjustment parameter. The flexibility of adjusting the photo can be improved.

Description

Photo adjusting method and device, storage medium and electronic equipment
Technical Field
The present application belongs to the field, and in particular, to a method and an apparatus for adjusting a photo, a storage medium, and an electronic device.
Background
Cameras are installed on a plurality of intelligent terminals, and comprise a front camera and a rear camera, and pixels of the cameras can reach the level of ten million pixels. Users often take pictures using the terminal. In addition to the requirement of sufficiently clear pictures, users have increasingly demanding beautification requirements for pictures. In the related art, the terminal may perform adjustment based on a color histogram and a color space transformation method on a picture taken by a user. However, these photo adjustment methods have poor flexibility.
Disclosure of Invention
The embodiment of the application provides a photo adjusting method and device, a storage medium and an electronic device, which can improve the flexibility of adjusting photos.
The embodiment of the application provides a method for adjusting a photo, which comprises the following steps:
acquiring a photo to be adjusted;
performing semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model to obtain a photo segmentation result;
determining the local object category contained in the photo to be adjusted according to the photo segmentation result;
and acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type by using each target adjustment parameter.
The embodiment of the application provides an adjusting device of photo, includes:
the acquisition module is used for acquiring a photo to be adjusted;
the photo segmentation module is used for performing semantic segmentation on the photo to be adjusted by utilizing a preset image semantic segmentation model to obtain a photo segmentation result;
the determining module is used for determining the local object category contained in the photo to be adjusted according to the photo segmentation result;
and the adjusting module is used for acquiring a target adjusting parameter corresponding to each local object type and adjusting each local object type by using each target adjusting parameter.
The embodiment of the present application provides a storage medium, on which a computer program is stored, and when the computer program is executed on a computer, the computer is caused to execute the steps in the adjusting method of the photo provided by the embodiment of the present application.
The embodiment of the present application further provides an electronic device, which includes a memory and a processor, where the processor is configured to execute the steps in the method for adjusting a photo provided in the embodiment of the present application by calling the computer program stored in the memory.
The terminal can firstly carry out semantic segmentation on the photo to be adjusted, and then determines the local object type contained in the photo to be adjusted according to a semantic segmentation result. Then, the terminal may obtain target adjustment parameters corresponding to the respective local object categories, and individually adjust the respective local object categories using the respective target adjustment parameters. Therefore, the present embodiment can individually adjust each local object in the photo that needs to be adjusted, thereby improving flexibility of adjusting the polarity of the photo.
Drawings
The technical solution and the advantages of the present invention will be apparent from the following detailed description of the embodiments of the present invention with reference to the accompanying drawings.
Fig. 1 is a schematic flowchart of a photo adjustment method according to an embodiment of the present application.
Fig. 2 is another schematic flowchart of a photo adjustment method according to an embodiment of the present application.
Fig. 3 to fig. 4 are scene schematic diagrams of an adjusting method of a photo according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of an adjusting apparatus for photos according to an embodiment of the present application.
Fig. 6 is another schematic structural diagram of an apparatus for adjusting a photograph according to an embodiment of the present application.
Fig. 7 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application.
Fig. 8 is another schematic structural diagram of a mobile terminal according to an embodiment of the present application.
Detailed Description
Referring to the drawings, wherein like reference numbers refer to like elements, the principles of the present invention are illustrated as being implemented in a suitable computing environment. The following description is based on illustrated embodiments of the invention and should not be taken as limiting the invention with regard to other embodiments that are not detailed herein.
It can be understood that the execution subject of the embodiment of the present application may be a terminal device such as a smart phone or a tablet computer.
Referring to fig. 1, fig. 1 is a schematic flow chart of a photo adjustment method according to an embodiment of the present application, where the flow chart may include:
in step S101, a photograph to be adjusted is acquired.
In step S102, a preset image semantic segmentation model is used to perform semantic segmentation on the photo to be adjusted, so as to obtain a photo segmentation result.
Cameras are installed on a plurality of intelligent terminals, and comprise a front camera and a rear camera, and pixels of the cameras can reach the level of ten million pixels. Users often take pictures using the terminal. In addition to the requirement of sufficiently clear pictures, users have increasingly demanding beautification requirements for pictures. In the related art, the terminal may perform adjustment based on a color histogram and a color space transformation method on a picture taken by a user. However, these photo adjustment methods can only make global beautification on photos, and their flexibility is poor.
In step S101 of the embodiment of the present application, for example, the terminal may first obtain a photo that needs to be beautified, that is, a photo to be adjusted.
Then, the terminal can perform semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model, and obtain a semantic segmentation result of the photo.
The semantic division of the photo means that the terminal can automatically divide and recognize the content in the photo. For example, a photo of a user riding a motorcycle is input into the preset image semantic segmentation model, and then the output of the preset image semantic segmentation model should be able to label the regions of the person, the motorcycle, and the background in the photo respectively. For example, the terminal can label the region of the person in the photo with red, label the region of the motorcycle with green, and label the background with black by using a preset image semantic segmentation model.
The preset image semantic segmentation model can adopt a full convolution neural network (FCN), deep lab and the like.
In step S103, the local object class included in the photo to be adjusted is determined according to the photo segmentation result.
For example, after obtaining the semantic segmentation result of the photo to be adjusted, the terminal may determine the local object category included in the photo to be adjusted according to the semantic segmentation result. That is, the terminal may determine the local object class included in the semantically segmented photograph.
It should be noted that the local object category refers to objects belonging to different categories that exist in the photograph. For example, the photograph to be adjusted is a scene of a user riding a motorcycle. Then, at least the following three local object categories are included in the photograph to be adjusted: characters, motorcycles, and backgrounds.
In step S104, a target adjustment parameter corresponding to each of the local object categories is obtained, and each of the local object categories is adjusted by using each of the target adjustment parameters.
For example, after determining the types of the local objects included in the photo to be adjusted, the terminal may obtain an adjustment parameter corresponding to each of the types of the local objects, that is, a target adjustment parameter. Then, the terminal can adjust each local object type corresponding to each target adjustment parameter by using each target adjustment parameter, so as to obtain an adjusted photo.
For example, the photo to be adjusted includes three local object categories of a person, a motorcycle, and a background, and then the terminal may sequentially obtain a first target adjustment parameter corresponding to the person, a second target adjustment parameter corresponding to the motorcycle, and a third target adjustment parameter corresponding to the background. Then, the terminal can respectively use the first target adjustment parameter to adjust the area where the person (user) is located in the photo, use the second target adjustment parameter to adjust the area where the motorcycle is located in the photo, and use the third target adjustment parameter to adjust the area where the background is located in the photo.
It can be understood that, in the embodiment of the present application, the terminal may perform semantic segmentation on the photo to be adjusted first, and then determine the local object category included in the photo to be adjusted according to the semantic segmentation result. Then, the terminal may obtain target adjustment parameters corresponding to the respective local object categories, and individually adjust the respective local object categories using the respective target adjustment parameters. Therefore, the present embodiment can individually adjust each local object in the photo that needs to be adjusted, thereby improving the flexibility of adjusting the photo.
Referring to fig. 2, fig. 2 is another schematic flow chart of a photo adjustment method according to an embodiment of the present application, where the flow chart may include:
in step S201, the terminal acquires a photo to be adjusted.
In step S202, the terminal performs semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model, so as to obtain a photo segmentation result.
For example, steps S201 and S202 may include:
after the camera shoots and obtains the picture, the terminal can firstly obtain the picture which needs to be beautified, namely the picture to be adjusted.
Then, the terminal can perform semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model, and obtain a semantic segmentation result of the photo.
The semantic division of the photo means that the terminal can automatically divide and recognize the content in the photo. For example, a photo of a user riding a motorcycle is input into the preset image semantic segmentation model, and then the output of the preset image semantic segmentation model should be able to label the regions of the person, the motorcycle, and the background in the photo respectively. For example, the terminal can label the region of the person in the photo with red, label the region of the motorcycle with green, and label the background with black by using a preset image semantic segmentation model.
The preset image semantic segmentation model can adopt a full convolution neural network (FCN), deep lab and the like.
In step S203, according to the photo segmentation result, the terminal determines the local object class included in the photo to be adjusted.
For example, after obtaining the semantic segmentation result of the photo to be adjusted, the terminal may determine the local object category included in the photo to be adjusted according to the semantic segmentation result. That is, the terminal may determine the local object class included in the semantically segmented photograph.
It should be noted that the local object category refers to objects belonging to different categories that exist in the photograph. For example, the photograph to be adjusted is a scene of a user riding a motorcycle. Then, at least the following three local object categories are included in the photograph to be adjusted: characters, motorcycles, and backgrounds.
In step S204, the terminal counts the number of the local object categories.
For example, after determining the local object classes contained in the semantic segmentation result of the photo to be adjusted, the terminal may count the number of the local object classes.
Then, the terminal may detect whether the number of the local object categories reaches a preset first threshold.
If it is detected that the number of the types of the local objects does not reach the preset first threshold, the photo to be adjusted can be regarded as the photo taken in the simple scene, and at this time, the terminal may not perform additional adjustment on the photo, and so on.
If it is detected that the number of the local object categories reaches the preset first threshold, the process proceeds to step S205.
In step S205, if it is detected that the number reaches the preset first threshold, the terminal obtains a target adjustment parameter corresponding to each of the local object categories, and adjusts each of the local object categories by using each of the target adjustment parameters.
For example, the terminal counts that the number of the local object categories included in the photo to be adjusted reaches a preset first threshold, and at this time, the photo to be adjusted may be considered to be a photo taken in a complex scene.
It should be noted that the complex scene refers to a shot scene of a photo that includes a plurality of local object classes. For example, a photograph taken in an outdoor environment including a person includes various objects such as buildings, plants, cars, roads, bicycles, and the like in addition to the person in the scene, and it can be considered that the photograph is taken in a complicated scene.
In this case, the terminal may be triggered to acquire the adjustment parameter corresponding to each local object class, i.e., the target adjustment parameter. Then, the terminal can adjust each local object type corresponding to each target adjustment parameter by using each target adjustment parameter, so as to obtain an adjusted photo.
For example, the photo to be adjusted includes three local object categories of a person, a motorcycle, and a background, and the value of the preset first threshold is 3, that is, the number of the local object categories reaches the preset first threshold. Then, the terminal may sequentially acquire a first target adjustment parameter corresponding to the character, a second target adjustment parameter corresponding to the motorcycle, and a third target adjustment parameter corresponding to the background. Then, the terminal can respectively use the first target adjustment parameter to adjust the area where the person (user) is located in the photo, use the second target adjustment parameter to adjust the area where the motorcycle is located in the photo, and use the third target adjustment parameter to adjust the area where the background is located in the photo.
In one embodiment, each target adjustment parameter may be a parameter for adjusting color. It will be appreciated that the color of the photograph may become more natural and full after the photograph is adjusted using the color-related tuning parameters.
In step S206, when it is detected that the user browses the adjusted photos, the terminal acquires facial expression features of the user.
For example, after adjusting the photo to be adjusted by using each target adjustment parameter, the terminal detects that the user browses the adjusted photo. For example, the terminal detects that the user enters an album from a shooting preview interface of the camera to browse the photos just taken. At this time, the terminal may be triggered to acquire a face image of the user, and the facial expression features of the user may be acquired according to the face image.
For example, the terminal may start a front-facing camera to acquire a face image of the user, and acquire facial expression features of the user according to the face image.
After the facial expression features of the user are acquired, the terminal can analyze the facial expression features to judge whether the user is in a preset negative expression state. In some embodiments, the preset negative expression state may be a relatively negative expression state such as a bow, pucker, disappointment, or the like. In contrast, smiling, mouth-opening, pleasure, etc. are relatively positive expression states.
If the user is analyzed to be in a more positive expression state according to the facial expression characteristics of the user, the user can be considered to be satisfied with the currently browsed and adjusted photo, and at the moment, the terminal can execute other operations.
If it is analyzed that the user is in a state of having a negative expression, such as a bow, disappointment, etc., based on the facial expression characteristics of the user, the process proceeds to step S207.
In step S207, if it is determined that the user is in the preset negative expression state according to the facial expression feature, the terminal detects whether the photo is deleted within the adjusted preset duration.
In step S208, if it is detected that the photo is deleted within the adjusted preset duration, the terminal determines that the photo segmentation result has an error, and counts the number of times that the photo segmentation result has the error.
In step S209, when it is detected that the number of times reaches a preset second threshold, the terminal replaces the preset image semantic segmentation model.
For example, steps S207, S208, and S209 may include:
the terminal analyzes that the user is in a relatively negative expression when browsing the adjusted photo according to the acquired facial expression characteristics of the user, so that the user can be considered to feel dissatisfied with the adjusted photo, and at the moment, the terminal can detect whether the photo is deleted within the adjusted preset duration.
If the photo is detected not to be deleted within the adjusted preset time length, the terminal can execute other operations.
If it is detected that the photo is deleted within the adjusted preset time period, for example, the photo is deleted by the user within 3 seconds after the adjustment, the user may be considered to delete the photo because the adjusted photo is not satisfactory. In this case, it is likely that the effect is not good when the local object class is adjusted later using the target adjustment parameter because the erroneous image segmentation is performed when the semantic segmentation is performed on the photograph to be adjusted. At this time, the terminal may determine that the photo semantic segmentation result has an error. Meanwhile, the terminal can count the times of errors in the segmentation result when the preset image semantic segmentation model configured for the terminal is used for carrying out photo semantic segmentation.
Then, the terminal may detect whether the number of times reaches a preset second threshold.
If the number of times is detected to be smaller than the preset second threshold, it can be considered that the number of errors occurring when the preset image semantic segmentation model configured for the terminal performs photo semantic segmentation is still small, and at this time, the terminal can perform other operations.
If the number of times reaches the preset second threshold value, it can be considered that the preset image semantic segmentation model currently configured for the terminal has more errors during photo semantic segmentation. In this case, the terminal may replace the preset image semantic segmentation model. For example, the image semantic segmentation model previously configured for the terminal is the A model. Then, when the number of times of errors in the result of semantic segmentation of the photo by using the A model is detected to reach a preset second threshold value, the terminal can acquire another image semantic segmentation model. For example, the terminal may obtain a b-model and replace the a-model with the b-model.
In an implementation, this embodiment may further include the following steps:
the method comprises the steps that a terminal obtains the number of local object types contained in each picture needing to be adjusted and shot within a preset time range;
according to the number of the local object types contained in each photo, the terminal calculates a corresponding average value;
and updating the preset first threshold value by the terminal according to the average value.
For example, in step S204, the terminal may count the number of local object categories included in the photo to be adjusted. Then, the terminal may record the number of local object categories included in each picture that needs to be adjusted and is taken within a preset time range, for example, within the last week.
Then, the terminal may calculate an average value of the recorded numbers of the local object categories included in the respective photos, and update the value of the preset first threshold according to the average value.
For example, the terminal may set the average value to a preset first threshold value. Or, the terminal may also increase or decrease a certain amplitude based on the average value to obtain a target value, and set the target value as the preset first threshold.
Referring to fig. 3 to 4, fig. 3 to 4 are schematic scene diagrams of a photo adjustment method according to an embodiment of the present disclosure.
In one embodiment, the trained image semantic segmentation model can be transplanted into the terminal. The preset image semantic segmentation model can be obtained by the following steps: first, the machine may take a large number of photographs containing various photographed scenes, including various photographed objects, such as people, buildings, various vehicles, various tables and chairs, and so on. The machine may then perform pixel-level object targeting on each photograph with human assistance. Then, the terminal can obtain a preselected image semantic segmentation model, and input the picture subjected to pixel-level object calibration as a training sample into the image semantic segmentation model, so that deep learning training is performed on the picture to obtain a trained image semantic segmentation model, and the trained image semantic segmentation model is transplanted into the terminal. After the image semantic segmentation model is obtained, the terminal can determine the image semantic segmentation model as a preset image semantic segmentation model. It can be understood that, because the picture calibrated by the object at the pixel level is used as the training sample in the training process of the preset image semantic segmentation model, the segmentation precision of the preset image semantic segmentation model on the image is very high.
For example, when the user takes a photo nail, the terminal may acquire the photo nail to be beautified and determine it as the photo nail to be adjusted, for example, the photo nail is as shown in fig. 3.
Then, the terminal can perform semantic segmentation on the photo A by using a preset image semantic segmentation model, and obtain a semantic segmentation result of the photo A.
For example, as shown in fig. 4, the preset image semantic segmentation model segments the photo a into a person 10, a building 20, a cloud 30, and a background 40.
After obtaining the semantic segmentation result of the photo A, the terminal can determine the local object types contained in the photo A according to the semantic segmentation result, and count the number of the local object types. For example, the terminal determines that the photo A contains 4 local object categories of people, buildings, clouds and background.
Then, the terminal may detect whether the number of the local object categories included in the photo A reaches a preset first threshold. For example, the preset first threshold is 3. Then, the terminal may detect that the number of local object classes contained in the photo A exceeds a preset first threshold. In this case, the photo a is a photo taken in a complex scene, and individual color adjustment needs to be performed for each local object type.
In this case, the terminal may be triggered to acquire the adjustment parameter corresponding to each local object class, i.e., the target adjustment parameter. Then, the terminal can adjust each local object type corresponding to each target adjustment parameter by using each target adjustment parameter, so as to obtain an adjusted photo.
For example, the terminal may sequentially acquire a first target adjustment parameter corresponding to a character, a second target adjustment parameter corresponding to a building, a third target adjustment parameter corresponding to a cloud, and a fourth target adjustment parameter corresponding to a background. Then, the terminal can respectively use the first target adjustment parameter to adjust the region where the person is located in the photo, use the second target adjustment parameter to adjust the region where the building is located in the photo, use the third target adjustment parameter to adjust the region where the cloud is located in the photo, and use the fourth target adjustment parameter to adjust the region where the background is located in the photo.
It can be understood that after the local object categories in the photo album are respectively subjected to parameter tuning by using the target tuning parameters, the color of the photo album can become more natural and full.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an adjusting device for photos according to an embodiment of the present application. The photo adjusting apparatus 300 may include: an acquisition module 301, a photo segmentation module 302, a determination module 303, and an adjustment module 304.
An obtaining module 301, configured to obtain a photo to be adjusted.
The photo segmentation module 302 is configured to perform semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model to obtain a photo segmentation result.
For example, the obtaining module 301 may obtain a photo that needs to be beautified, i.e. a photo to be adjusted.
Then, the photo segmentation module 302 may perform semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model, and obtain a semantic segmentation result of the photo.
The semantic division of the photo means that the terminal can automatically divide and recognize the content in the photo. For example, a photo of a user riding a motorcycle is input into the preset image semantic segmentation model, and then the output of the preset image semantic segmentation model should be able to label the regions of the person, the motorcycle, and the background in the photo respectively. For example, the terminal can label the region of the person in the photo with red, label the region of the motorcycle with green, and label the background with black by using a preset image semantic segmentation model.
The preset image semantic segmentation model can adopt a full convolution neural network (FCN), deep lab and the like.
A determining module 303, configured to determine a local object category included in the photo to be adjusted according to the photo segmentation result.
For example, after obtaining the semantic segmentation result of the photo to be adjusted, the determining module 303 may determine the local object category included in the photo to be adjusted according to the semantic segmentation result. That is, the determining module 303 may determine the local object class included in the semantically segmented photograph.
It should be noted that the local object category refers to objects belonging to different categories that exist in the photograph. For example, the photograph to be adjusted is a scene of a user riding a motorcycle. Then, at least the following three local object categories are included in the photograph to be adjusted: characters, motorcycles, and backgrounds.
The adjusting module 304 is configured to obtain a target adjustment parameter corresponding to each local object category, and adjust each local object category by using each target adjustment parameter.
For example, after the determining module 303 determines the local object categories included in the photo to be adjusted, the adjusting module 304 may obtain an adjusting parameter corresponding to each local object category, that is, a target adjusting parameter. Then, the adjusting module 304 may adjust the corresponding local object categories by using the target adjustment parameters, so as to obtain the adjusted photos.
For example, if the photo to be adjusted includes three local object categories of a person, a motorcycle, and a background, the adjusting module 304 may sequentially obtain a first target adjustment parameter corresponding to the person, a second target adjustment parameter corresponding to the motorcycle, and a third target adjustment parameter corresponding to the background. Then, the adjusting module 304 may respectively use the first target adjustment parameter to adjust the region where the person (user) is located in the photo, use the second target adjustment parameter to adjust the region where the motorcycle is located in the photo, and use the third target adjustment parameter to adjust the region where the background is located in the photo.
Referring to fig. 6, fig. 6 is another schematic structural diagram of an adjusting apparatus for photos according to an embodiment of the present disclosure. In an embodiment, the photo adjusting apparatus 300 may further include: a statistics module 305, a replacement module 306, a detection module 307, and an update module 308.
A counting module 305, configured to count the number of the local object categories.
Then, the adjustment module 304 is configured to: and if the quantity is detected to reach a preset first threshold value, acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type by using each target adjustment parameter.
For example, after the determining module 303 determines the local object categories included in the photo to be adjusted according to the semantic segmentation result of the photo, the counting module 305 may count the number of the local object categories.
Then, the terminal may detect whether the number of the local object categories reaches a preset first threshold.
If it is detected that the number of the types of the local objects does not reach the preset first threshold, the photo to be adjusted can be regarded as the photo taken in the simple scene, and at this time, the terminal may not perform additional adjustment on the photo, and so on.
If the terminal counts that the number of the local object types included in the photo to be adjusted reaches a preset first threshold value, the photo to be adjusted can be regarded as a photo obtained by shooting in a complex scene.
It should be noted that the complex scene refers to a shot scene of a photo that includes a plurality of local object classes. For example, a photograph taken in an outdoor environment including a person includes various objects such as buildings, plants, cars, roads, bicycles, and the like in addition to the person in the scene, and it can be considered that the photograph is taken in a complicated scene.
In this case, the adjustment module 304 may be triggered to obtain the adjustment parameters corresponding to each local object class, i.e., the target adjustment parameters. Then, the adjusting module 304 may adjust the corresponding local object categories by using the target adjustment parameters, so as to obtain the adjusted photos.
For example, the photo to be adjusted includes three local object categories of a person, a motorcycle, and a background, and the value of the preset first threshold is 3, that is, the number of the local object categories reaches the preset first threshold. Then, the adjusting module 304 may sequentially obtain a first target adjustment parameter corresponding to the person, a second target adjustment parameter corresponding to the motorcycle, and a third target adjustment parameter corresponding to the background. Then, the adjusting module 304 may respectively use the first target adjustment parameter to adjust the region where the person (user) is located in the photo, use the second target adjustment parameter to adjust the region where the motorcycle is located in the photo, and use the third target adjustment parameter to adjust the region where the background is located in the photo.
The replacing module 306 is configured to determine that an error exists in the photo segmentation result and count the number of times that the error exists in the photo segmentation result if it is detected that the photo is deleted within the adjusted preset duration; and when the times are detected to reach a preset second threshold value, replacing the preset image semantic segmentation model.
For example, after the adjusting module 304 adjusts the photos, the replacing module 306 can detect whether the adjusted photos are deleted within the adjusted preset time period.
If the photo is detected not to be deleted within the adjusted preset time length, the terminal can execute other operations.
If it is detected that the photo is deleted within the adjusted preset time period, for example, the photo is deleted by the user within 3 seconds after the adjustment, the user may be considered to delete the photo because the adjusted photo is not satisfactory. In this case, it is likely that the effect is not good when the local object class is adjusted later using the target adjustment parameter because the erroneous image segmentation is performed when the semantic segmentation is performed on the photograph to be adjusted. At this point, the replacement module 306 may determine that the photo semantic segmentation result is erroneous. Meanwhile, the replacing module 306 may count the number of times that the segmentation result has errors when performing the photo semantic segmentation by using the preset image semantic segmentation model currently configured for the terminal.
The replacement module 306 may then detect whether the number of times reaches a preset second threshold.
If the number of times is detected to be smaller than the preset second threshold, it can be considered that the number of errors occurring when the preset image semantic segmentation model configured for the terminal performs photo semantic segmentation is still small, and at this time, the terminal can perform other operations.
If the number of times reaches the preset second threshold value, it can be considered that the preset image semantic segmentation model currently configured for the terminal has more errors during photo semantic segmentation. In this case, the replacement module 306 may replace the preset image semantic segmentation model. For example, the image semantic segmentation model previously configured for the terminal is the A model. Then, when it is detected that the number of times of errors in the result of the semantic segmentation of the photo by using the a model reaches a preset second threshold, the replacing module 306 may obtain another image semantic segmentation model. For example, the replacement module 306 may obtain a B model and replace the A model with the B model.
The detection module 307 is configured to, when it is detected that the user browses the adjusted photo, obtain a facial expression feature of the user; and if the user is determined to be in a preset negative expression state according to the facial expression characteristics, detecting whether the photo is deleted within the adjusted preset time length.
For example, after the adjusting module 304 adjusts the photo to be adjusted by using the target adjusting parameters, the terminal detects that the user browses the adjusted photo. For example, the detection module 307 detects that the user enters the photo album from the camera's capture preview interface to view the just captured photo. At this time, the detection module 307 may be triggered to acquire a facial image of the user, and acquire facial expression features of the user according to the facial image.
For example, the detection module 307 may turn on a front camera to obtain a facial image of the user, and obtain facial expression features of the user according to the facial image.
After obtaining the facial expression features of the user, the detection module 307 may analyze the facial expression features to determine whether the user is in a preset negative expression state. In some embodiments, the preset negative expression state may be a relatively negative expression state such as a bow, pucker, disappointment, or the like. In contrast, smiling, mouth-opening, pleasure, etc. are relatively positive expression states.
If the user is analyzed to be in a more positive expression state according to the facial expression characteristics of the user, the user can be considered to be satisfied with the currently browsed and adjusted photo, and at the moment, the terminal can execute other operations.
If the detection module 307 analyzes that the user has a relatively negative expression when browsing the adjusted photo according to the obtained facial expression features of the user, it may be considered that the user feels dissatisfied with the adjusted photo, and at this time, the terminal may detect whether the photo is deleted within the adjusted preset duration.
If the photo is detected not to be deleted within the adjusted preset time length, the terminal can execute other operations.
If the photo is detected to be deleted within the adjusted preset time length, the replacing module 306 may determine that the photo semantic segmentation result has an error.
An updating module 308, configured to obtain the number of local object categories included in each picture that needs to be adjusted and is taken within a preset time range; calculating a corresponding average value according to the number of the local object categories contained in each photo; and updating the preset first threshold according to the average value.
For example, the counting module 305 counts the number of local object classes contained in the photo to be adjusted. Then, the update module 308 may record the number of local object categories included in each picture that needs to be adjusted and is taken within a predetermined time range, such as the last week.
Then, the updating module 308 may calculate an average value of the recorded numbers of the local object categories included in the respective photos, and update the value of the preset first threshold according to the average value.
For example, the update module 308 may set the average to a preset first threshold. Alternatively, the updating module 308 may also increase or decrease a certain amplitude based on the average value to obtain a target value, and set the target value as the preset first threshold.
The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed on a computer, the computer is caused to execute the steps in the photo adjustment method provided in this embodiment.
The embodiment of the present application further provides an electronic device, which includes a memory and a processor, where the processor is configured to execute the steps in the method for adjusting a photo provided in this embodiment by calling a computer program stored in the memory.
For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Referring to fig. 7, fig. 7 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application.
The mobile terminal 400 may include a camera unit 401, a memory 402, a processor 403, and the like. Those skilled in the art will appreciate that the mobile terminal architecture shown in fig. 7 is not intended to be limiting of mobile terminals and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
The camera unit 401 may include a front camera, a rear camera, and the like.
The memory 402 may be used to store applications and data. The memory 402 stores applications containing executable code. The application programs may constitute various functional modules. The processor 403 executes various functional applications and data processing by running an application program stored in the memory 402.
The processor 403 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by running or executing an application program stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the mobile terminal.
In this embodiment, the processor 403 in the mobile terminal loads the executable code corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 403 runs the application programs stored in the memory 402, thereby implementing the steps:
acquiring a photo to be adjusted; performing semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model to obtain a photo segmentation result; determining the local object category contained in the photo to be adjusted according to the photo segmentation result; and acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type by using each target adjustment parameter.
Referring to fig. 8, the mobile terminal 500 may include a camera unit 501, a memory 502, a processor 503, an input unit 504, an output unit 505, a speaker 506, a microphone 507, and the like.
The camera unit 501 may include a front camera, a rear camera, and the like.
The memory 502 may be used to store applications and data. Memory 502 stores applications containing executable code. The application programs may constitute various functional modules. The processor 503 executes various functional applications and data processing by running an application program stored in the memory 502.
The processor 503 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by running or executing an application program stored in the memory 502 and calling data stored in the memory 502, thereby performing overall monitoring of the mobile terminal.
The input unit 504 may be used to receive input numbers, character information, or user characteristic information (such as a fingerprint), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.
The output unit 505 may be used to display information input by or provided to a user and various graphic user interfaces of the mobile terminal, which may be configured by graphics, text, icons, video, and any combination thereof. The output unit may include a display panel.
In this embodiment, the processor 503 in the mobile terminal loads the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 runs the application programs stored in the memory 502, thereby implementing the steps:
acquiring a photo to be adjusted; performing semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model to obtain a photo segmentation result; determining the local object category contained in the photo to be adjusted according to the photo segmentation result; and acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type by using each target adjustment parameter.
In one embodiment, after the step of determining the local object class included in the photo to be adjusted is performed, the processor 503 may further perform: and counting the number of the local object categories.
Then, when the step of obtaining the target adjustment parameter corresponding to each of the local object categories and adjusting each of the local object categories by using each of the target adjustment parameters is executed, the processor 503 may perform: and if the quantity is detected to reach a preset first threshold value, acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type by using each target adjustment parameter.
In one embodiment, after performing the step of adjusting each local object class by using each target adjustment parameter, the processor 503 may further perform: if the photo is detected to be deleted within the adjusted preset time length, determining that the photo segmentation result has errors, and counting the times of the photo segmentation result having errors; and when the times are detected to reach a preset second threshold value, replacing the preset image semantic segmentation model.
In an embodiment, before performing the step of determining that the photo segmentation result has an error if it is detected that the photo is deleted within the adjusted preset time duration, the processor 503 may further perform: when it is detected that the user browses the adjusted photo, acquiring facial expression characteristics of the user; and if the user is determined to be in a preset negative expression state according to the facial expression characteristics, detecting whether the photo is deleted within the adjusted preset time length.
In one embodiment, the processor 503 may further perform the following steps: acquiring the number of local object types contained in each picture needing to be adjusted and shot within a preset time range; calculating a corresponding average value according to the number of the local object categories contained in each photo; and updating the preset first threshold according to the average value.
In the above embodiments, the descriptions of the embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed description of the adjustment method for the photo, and are not described herein again.
The photo adjusting device provided in the embodiment of the present application and the photo adjusting method in the above embodiment belong to the same concept, and any method provided in the photo adjusting method embodiment may be run on the photo adjusting device, and a specific implementation process thereof is described in the photo adjusting method embodiment in detail, and is not described herein again.
It should be noted that, for the photo adjustment method described in the embodiment of the present application, it can be understood by those skilled in the art that all or part of the process for implementing the photo adjustment method described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, where the computer program can be stored in a computer readable storage medium, such as a memory, and executed by at least one processor, and during the execution process, the process of the embodiment of the photo adjustment method can be included. The storage medium may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like.
In the photo adjusting device according to the embodiment of the present application, each functional module may be integrated into one processing chip, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium, such as a read-only memory, a magnetic or optical disk, or the like.
The method, the apparatus, the storage medium, and the electronic device for adjusting a photo provided in the embodiments of the present application are described in detail above, and a specific example is applied in the present application to explain the principle and the implementation of the present invention, and the description of the above embodiments is only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (8)

1. A method for adjusting a photograph, comprising:
acquiring a photo to be adjusted;
performing semantic segmentation on the photo to be adjusted by using a preset image semantic segmentation model to obtain a photo segmentation result;
determining all local object categories contained in the photo to be adjusted according to the photo segmentation result, wherein the local object categories comprise one or more of a human body category, a background category and a plurality of object categories;
counting the number of the local object categories;
if the number reaches a preset first threshold value, judging that the photo contains a complex scene, acquiring a target adjustment parameter corresponding to each local object type, and adjusting each local object type in the multiple local object types by using each target adjustment parameter;
and if the number is detected not to reach a preset first threshold value, judging that the photo contains a simple scene, and not adjusting the photo.
2. The method for adjusting a photo according to claim 1, wherein after the step of adjusting each of the local object categories in each of the local object categories by using each of the target adjustment parameters, the method further comprises:
if the photo is detected to be deleted within the adjusted preset time length, determining that the photo segmentation result has errors, and counting the times of the photo segmentation result having errors;
and when the times are detected to reach a preset second threshold value, replacing the preset image semantic segmentation model.
3. The method for adjusting a photo according to claim 2, wherein before the step of determining that the photo segmentation result has an error if it is detected that the photo is deleted within the adjusted preset duration, the method further comprises:
when it is detected that the user browses the adjusted photo, acquiring facial expression characteristics of the user;
and if the user is determined to be in a preset negative expression state according to the facial expression characteristics, detecting whether the photo is deleted within the adjusted preset time length.
4. The method for adjusting a photograph according to claim 3, wherein the method further comprises:
acquiring the number of a plurality of local object categories contained in each picture needing to be adjusted and shot within a preset time range;
calculating a corresponding average value according to the number of the local object categories contained in each photo;
and updating the preset first threshold according to the average value.
5. An apparatus for adjusting a photograph, comprising:
the acquisition module is used for acquiring a photo to be adjusted;
the photo segmentation module is used for performing semantic segmentation on the photo to be adjusted by utilizing a preset image semantic segmentation model to obtain a photo segmentation result;
the determining module is used for determining all local object categories contained in the photo to be adjusted according to the photo segmentation result;
the statistical module is used for counting the number of the local object categories;
and the adjusting module is used for judging that the photo contains a complex scene if the quantity is detected to reach a preset first threshold value, acquiring a target adjusting parameter corresponding to each local object type, and adjusting each local object type in the multiple local object types by using each target adjusting parameter respectively, and judging that the photo contains a simple scene if the quantity is detected not to reach the preset first threshold value, and not adjusting the photo.
6. The apparatus for adjusting a photograph according to claim 5, wherein the apparatus further comprises: replacement module for
If the photo is detected to be deleted within the adjusted preset time length, determining that the photo segmentation result has errors, and counting the times of the photo segmentation result having errors;
and when the times are detected to reach a preset second threshold value, replacing the preset image semantic segmentation model.
7. A storage medium having stored thereon a computer program, characterized in that the computer program, when executed on a computer, causes the computer to execute the method according to any of claims 1 to 4.
8. An electronic device comprising a memory, a processor, wherein the processor is configured to perform the method of any of claims 1 to 4 by invoking a computer program stored in the memory.
CN201711407286.5A 2017-12-22 2017-12-22 Photo adjusting method and device, storage medium and electronic equipment Active CN109961403B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201711407286.5A CN109961403B (en) 2017-12-22 2017-12-22 Photo adjusting method and device, storage medium and electronic equipment
PCT/CN2018/116746 WO2019120025A1 (en) 2017-12-22 2018-11-21 Photograph adjustment method and apparatus, storage medium and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711407286.5A CN109961403B (en) 2017-12-22 2017-12-22 Photo adjusting method and device, storage medium and electronic equipment

Publications (2)

Publication Number Publication Date
CN109961403A CN109961403A (en) 2019-07-02
CN109961403B true CN109961403B (en) 2021-11-16

Family

ID=66993033

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711407286.5A Active CN109961403B (en) 2017-12-22 2017-12-22 Photo adjusting method and device, storage medium and electronic equipment

Country Status (2)

Country Link
CN (1) CN109961403B (en)
WO (1) WO2019120025A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111583283B (en) * 2020-05-20 2023-06-20 抖音视界有限公司 Image segmentation method, device, electronic equipment and medium
CN112040180B (en) * 2020-08-19 2022-06-07 浙江大华技术股份有限公司 Control method, device and system of image acquisition device
CN113283538B (en) * 2021-06-11 2022-07-22 长江大学 Insect classification and counting method based on convolutional neural network
CN114022496B (en) * 2021-09-26 2023-04-07 天翼爱音乐文化科技有限公司 Image processing method, system, device and storage medium

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010108475A (en) * 2008-10-03 2010-05-13 Sony Corp Image processing apparatus and method, program, and recording medium
CN101403949A (en) * 2008-11-11 2009-04-08 珠海金山软件股份有限公司 Hand-written input equipment and method
CN104685540B (en) * 2013-09-27 2017-04-26 华为技术有限公司 Image semantic segmentation method and apparatus
CN104935802B (en) * 2014-03-17 2018-02-27 联想(北京)有限公司 Image processing method, electronic equipment and system
CN104063842A (en) * 2014-05-30 2014-09-24 小米科技有限责任公司 Image processing method and device and terminal
CN104503749B (en) * 2014-12-12 2017-11-21 广东欧珀移动通信有限公司 Photo processing method and electronic equipment
CN105872350A (en) * 2015-12-08 2016-08-17 乐视移动智能信息技术（北京）有限公司 Adjusting method and device for photographing parameter of camera
CN105354565A (en) * 2015-12-23 2016-02-24 北京市商汤科技开发有限公司 Full convolution network based facial feature positioning and distinguishing method and system
CN105744171B (en) * 2016-03-30 2019-03-08 联想(北京)有限公司 A kind of image processing method and electronic equipment
CN106251369B (en) * 2016-07-22 2019-04-16 北京小米移动软件有限公司 Image processing method and device
CN106534687A (en) * 2016-11-18 2017-03-22 广东欧珀移动通信有限公司 Control method and control device
CN107358241B (en) * 2017-06-30 2021-01-26 Oppo广东移动通信有限公司 Image processing method, image processing device, storage medium and electronic equipment

Also Published As

Publication number Publication date
WO2019120025A1 (en) 2019-06-27
CN109961403A (en) 2019-07-02

Similar Documents

Publication Publication Date Title
CN109961403B (en) Photo adjusting method and device, storage medium and electronic equipment
CN107820020A (en) Method of adjustment, device, storage medium and the mobile terminal of acquisition parameters
CN108701439B (en) Image display optimization method and device
CN107909569B (en) Screen-patterned detection method, screen-patterned detection device and electronic equipment
CN112840376A (en) Image processing method, device and equipment
WO2021046715A1 (en) Exposure time calculation method, device, and storage medium
CN111209970A (en) Video classification method and device, storage medium and server
CN107330859B (en) Image processing method and device, storage medium and terminal
CN111835982B (en) Image acquisition method, image acquisition device, electronic device, and storage medium
CN108961267B (en) Picture processing method, picture processing device and terminal equipment
CN112532882B (en) Image display method and device
KR102545408B1 (en) Method and apparatus for detecting occluded image and medium
CN110889379A (en) Expression package generation method and device and terminal equipment
CN109784327B (en) Boundary box determining method and device, electronic equipment and storage medium
CN113411498A (en) Image shooting method, mobile terminal and storage medium
CN112200844A (en) Method, device, electronic equipment and medium for generating image
CN110717452B (en) Image recognition method, device, terminal and computer readable storage medium
WO2015196681A1 (en) Picture processing method and electronic device
CN112887615B (en) Shooting method and device
CN111316628A (en) Image shooting method and image shooting system based on intelligent terminal
CN107180417B (en) Photo processing method and device, computer readable storage medium and electronic equipment
CN111669492A (en) Method for processing shot digital image by terminal and terminal
CN107155000B (en) Photographing behavior analysis method and device and mobile terminal
CN108763491B (en) Picture processing method and device and terminal equipment
CN112672055A (en) Photographing method, device and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18

Applicant after: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd.

Address before: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18

Applicant before: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd.

GR01 Patent grant
GR01 Patent grant