CN109472795A - A kind of image edit method and device - Google Patents
A kind of image edit method and device Download PDFInfo
- Publication number
- CN109472795A CN109472795A CN201811265226.9A CN201811265226A CN109472795A CN 109472795 A CN109472795 A CN 109472795A CN 201811265226 A CN201811265226 A CN 201811265226A CN 109472795 A CN109472795 A CN 109472795A
- Authority
- CN
- China
- Prior art keywords
- adjustment
- image
- user
- pose adjustment
- object instance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 230000008921 facial expression Effects 0.000 claims description 33
- 230000011218 segmentation Effects 0.000 claims description 10
- 238000013527 convolutional neural network Methods 0.000 claims description 8
- 238000005096 rolling process Methods 0.000 claims description 8
- 230000009471 action Effects 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 238000005520 cutting process Methods 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- 238000013507 mapping Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 3
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 210000001508 eye Anatomy 0.000 description 1
- 238000005429 filling process Methods 0.000 description 1
- 238000009415 formwork Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/62—Control of parameters via user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
The present invention provides a kind of image edit method and devices, this method comprises: all examples for including in image to be edited are divided into instance objects independent；Pose adjustment is carried out to the object instance object that user selects according to the request of the pose adjustment of user；Background filling is carried out to the background absent region adjusted of object instance object gesture in image to be edited；Sprout wings to the Image Adjusting edge adjusted of object instance object gesture in image to be edited.The present invention can simplify picture editor's complexity.
Description
Technical field
The present invention relates to technical field of image processing, in particular to image edit method and device.
Background technique
In daily life, people always ceaselessly take the field of surrounding in order to leave souvenir or for sharing information
Scape, if but it is dissatisfied to the placement position of object, orientation when taking pictures, or certain objects is not intended to appear in photo, used
Then family can only be taken pictures again by moving position, moving object, very troublesome.
The prior art also provides some picture editing methods, can adjust to the primary attribute for photo of picture
Section, such as color value, brightness, transparency, but increasingly complex picture editor then needs professional and uses special figure
As processing software is handled, it is not particularly suited for ordinary user.
Summary of the invention
In view of this, picture editor can be simplified the purpose of the present invention is to provide a kind of image edit method and device
Complexity.
In order to achieve the above object, the present invention provides the following technical scheme that
A kind of image edit method, comprising:
All examples for including in image to be edited are divided into instance objects independent；
Pose adjustment is carried out to the object instance object that user selects according to the request of the pose adjustment of user；
Edge amendment is carried out to the adjustment edge adjusted of object instance object gesture in image to be edited.
A kind of image editing apparatus, comprising:
Example cutting unit, for all examples for including in image to be edited to be divided into example pair independent
As；
Pose adjustment unit, for carrying out appearance to the object instance object that user selects according to the request of the pose adjustment of user
State adjustment；
Edge amending unit, for carrying out side to the adjustment edge adjusted of object instance object gesture in image to be edited
Edge amendment.
As can be seen from the above technical solution, image edit method provided by the invention, by by the institute in image to be edited
There are instance objects to carry out example segmentation, posture is carried out to the object instance object that user selects according to the request of the pose adjustment of user
Adjustment, and exchange treating selvedge edge and be modified, it does not need user and deacclimatizes photographed scene, do not require user that there is the image of profession yet
Therefore processing technical ability can greatly simplify picture editting's complexity.
Detailed description of the invention
Fig. 1 is image edit method flow chart of the embodiment of the present invention；
Fig. 2 is realization process of the embodiment of the present invention using the instance objects in the identification of example network model and segmented image
Schematic diagram；
Fig. 3 is rolling target instance objects instance graph of the embodiment of the present invention；
Fig. 4 is mapping schematic diagram of the human object of the embodiment of the present invention to T-Pose model；
Fig. 5 is the process example figure that the embodiment of the present invention rejects object instance object；
Fig. 6 is deleted background filling process schematic diagram of the embodiment of the present invention；
Fig. 7 is the concrete application flow chart of one image edit method of the embodiment of the present invention；
Fig. 8 is the application result schematic diagram of one image edit method of the embodiment of the present invention；
Fig. 9 is the concrete application flow chart of two image edit method of the embodiment of the present invention；
Figure 10 is the application result schematic diagram of two image edit method of the embodiment of the present invention；
Figure 11 is the structural schematic diagram of image editing apparatus of the embodiment of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawing and according to embodiment,
Technical solution of the present invention is described in detail.
It is image edit method flow chart of the embodiment of the present invention referring to Fig. 1, Fig. 1, as shown in Figure 1, this method mainly includes
Following steps:
All examples for including in image to be edited are divided into instance objects independent by step 101.
Here, instance objects include human object (people i.e. in image) and non-human object (i.e. in image in addition to a person
Object, such as vehicle, trees, building etc.).
In order to divide the instance objects in image to be edited, needs first to identify each example in image to be edited, wrap
It includes and identifies each example occupied area and its classification (such as people, vehicle, tree, bird etc.), then the example identified is carried out again
The segmentation of Pixel-level example.Wherein the classification of example determines that the example is human object or non-human object.
For this purpose, pre-generated one can carry out the reality of identification and example segmentation to all kinds of examples in the embodiment of the present invention
Example network model carries out identification to all examples in image to be edited using the example network model and to each of identifying
Example carries out Pixel-level segmentation, to obtain the corresponding instance objects of each example in image to be edited, the example divided
It is mutually indepedent between object.
In the embodiment of the present invention, the example image of plurality of classes can be obtained in advance and is stored to example sample database, is passed through
All example images in example sample database are trained to obtain and above-mentioned can all kinds of examples be carried out with identification and example segmentation
Example network model.Wherein, all example images in example sample database are trained to obtain the side of example network model
Method is specific as follows:
To each example image in example sample database, following training operation is executed to obtain the example network model:
The characteristic pattern of the example image is extracted using convolutional neural networks (CNN) algorithm；
The object candidate area in the example image is extracted using region recommendation network (RPN)；
It is corrected pixel by ROIAlign algorithm, is aligned the pixel of object candidate area and characteristic pattern；
Divide and extract the area Mask for the instance objects for including in the example image based on full convolutional network (FCN) algorithm
Domain.
After training obtains examples detailed above network model, identification and example point can be carried out to the example in image to be edited
It cuts, using the realization process of the instance objects in example network model identification and segmented image as shown in Fig. 2, for including people
With the image of vehicle, walked by feature extraction conv convolution operation, RPN algorithm, ROIAlign algorithm, the full convolution algorithm of FCN four
Suddenly the classification (lclass), object candidate area (lbox) and the region mask (lmask) of each example are obtained, and based on this segmentation
The instance objects for the people for including in image out and the instance objects of vehicle.
Step 102 carries out pose adjustment to the object instance object that user selects according to the request of the pose adjustment of user.
In the present invention, the pose adjustment to non-human object includes: mobile, scaling, rotation.To the posture tune of human object
Divide exactly including except mobile, scaling and rotation, further including body posture adjustment and facial expression adjustment.
Wherein, for movement, scaling, rotate these three pose adjustments, either to human object or non-human object,
All use identical attitude adjusting method, wherein
When user requests mobile object instance object (i.e. the pose adjustment request of user is mobile object instance object),
Object instance object can be directly moved to the target position of the pose adjustment request instruction of user, it is real to reach mobile target
The purpose of example object；
When user requests scaling object instance object (the pose adjustment request of user is scaling object instance object)
When, it only need to be using object instance object centers position as origin, according to the scaling of the pose adjustment of user request instruction to mesh
Mark instance objects zoom in and out operation, and the purpose of scaling object instance object can be realized；
When user requests rolling target instance objects (i.e. the pose adjustment request of user is rolling target instance objects),
Can using object instance object centers position as origin, with the horizontal direction of the plane of image to be edited, vertical direction, perpendicular to
The direction of image to be edited is x-axis, y-axis and the z-axis of 3-D walls and floor, according to the target of the pose adjustment of user request instruction
Rotation angle of the instance objects on x-axis, y-axis and z-axis direction exists respectively to object instance object using 3D-GAN network model
Postrotational display area carries out estimating building on x-axis, y-axis and z-axis direction, and according to object instance object in x-axis, y-axis and z
The building result of estimating of three dimensions of axis determines display result of the postrotational object instance object in image to be edited.
Fig. 3 is rolling target instance objects exemplary diagram of the embodiment of the present invention, as shown in figure 3, to object instance object " vehicle "
Rotation process, the rotation being finally get translated into x-axis, y-axis and z-axis, thus object instance object " vehicle " respectively x-axis,
In y-axis and z-axis it is postrotational estimate building as a result, finally by these three estimate building result combine to obtain object instance pair
As " vehicle " postrotational display result.Here, it should be noted that, can not in original image in rolling target instance objects
It is directly obtained the structural information of some zone of ignorances after object instance object rotates, for this purpose, in the base of 3D-GAN network model
On plinth, by the angle of input object instance object current Pixel Information and rotation, to object instance object in three-dimensional side
Display area is estimated after upward adjustment, and zone of ignorance after rotating in three-dimensional coordinate to object instance object may be implemented
Calculating.
In the following, the realization side of the pose adjustment (body posture adjustment and facial expression adjustment) exclusive for human object
Method is introduced respectively:
In the present invention, either the body posture adjustment of object instance object or facial expression are adjusted, require elder generation
Object instance object is mapped based on skeleton (T-Pose) model, establishes the corresponding T-Pose mould of object instance object
Type, and identify the face key point of object instance object.Using the T-Pose model of object instance object to object instance object
Body posture adjustment and facial expression adjustment are carried out, pose adjustment process can be made to meet kinematic principle, each joint of linkage
Between movement.Human body instance objects to T-Pose model mapping as shown in figure 4, human body instance objects and corresponding T-Pose
It is corresponded between the key point of model.
Wherein,
Body posture is adjusted, following two method of adjustment is provided:
1, the body posture adjustment based on body posture template.
Under such method, need to be provided previously a variety of body posture templates.
User can select body posture template according to demand and trigger pose adjustment request, can take in pose adjustment request
Therefore body posture template with user's selection after receiving pose adjustment request, can use the object instance object
Object instance object is carried out body posture adjustment according to the body posture template by corresponding T-Pose model.
2, body posture is manually adjusted.
Under such method, user can the body posture directly to object instance object manually adjust, according to user
Manually adjust operation can determine adjustment position (such as head position, human arm, leg, hand) and adjust action message
(such as the information such as adjustment direction, distance, angle).The manual operation of user can also trigger pose adjustment request, and pose adjustment is asked
Middle carrying user is asked to manually adjust result information (i.e. adjustment position and adjustment action message), therefore, when receiving the posture tune
When whole request, can use the object instance object corresponding T-Pose model requested based on the pose adjustment in carry
It adjusts position and adjusts the body posture adjustment of action message.
In the embodiment of the present invention, facial expression is adjusted, is realized using facial expression template.
Facial expression adjustment includes the adjustment of eyes size, eyeball position, expression, the shape of the mouth as one speaks etc., it is also desirable to is first based on
Skeleton (T-Pose) model maps object instance object, establishes the corresponding T-Pose model of object instance object,
And identify the face key point of object instance object, to realize various facial expression adjustment in conjunction with face key point.
Furthermore it is possible to provide a variety of facial expression templates, user can select body posture template according to demand and trigger
Pose adjustment request can carry the facial expression template of user's selection in pose adjustment request, therefore, when receiving the posture tune
After whole request, it can determine the facial expression Template Information that the pose adjustment request of user carries, utilize the object instance object
Object instance object is carried out facial expression adjustment according to the facial expression template by corresponding T-Pose model.
In the embodiment of the present invention, body posture adjustment is being carried out to object instance object, that is, before executing this step 102,
It needs first to reject the object instance object that user selects from image to be edited, obtains the occupied area back of object instance object
The image to be edited of scape missing.The method for rejecting object instance object is: by the object instance of image to be edited and user's selection
Object carries out difference operation.
Fig. 5 shows the process example of a rejecting object instance object: in the instance objects being partitioned into image: " people "
After " vehicle ", user selects an object instance object, by the way that original image and the object instance image are carried out difference operation,
To reject object instance object from original image, such as select " people " for object instance object, then it will by difference operation
" people " rejects from original image, obtains the image of the background missing of " people " occupied area；Select " vehicle " for object instance object, then
" vehicle " is rejected from original image by difference operation, obtains the image of the background missing of " vehicle " occupied area.
In the embodiment of the present invention, body posture adjustment is being carried out to object instance object, that is, after executing this step 102,
It also needs further to carry out background filling to the background absent region adjusted of object instance object gesture in image to be edited.
It can use the network model based on the loss of duplex co content and local losses building, it is real-time to photo white space
Auto-complete carries out background filling to the background absent region adjusted of object instance object gesture in image to be edited, and background is filled out
Process is filled as shown in fig. 6, concrete implementation method is as follows:
Extract the global content characteristic figure of image to be edited (with reference to the global content forecast network model in Fig. 6)；
Extract the Local textural feature figure (with reference to the local grain network model in Fig. 6) on background absent region periphery；
According to the global content characteristic figure and Local textural feature figure, is lost based on confrontation and generate background absent region
Prognostic chart, and the prognostic chart is filled into background absent region.
Step 103 carries out edge amendment to the adjustment edge adjusted of object instance object gesture in image to be edited.
It is to the modified method in edge is carried out to the adjustment edge adjusted of object instance object gesture in image to be edited,
It exchanges treating selvedge edge and carries out feathering operation.
Image edit method of the embodiment of the present invention is described in detail above, below in conjunction with a specific example,
The application of the image edit method is illustrated.
In daily life, in order to leave souvenir or for sharing information, user always ceaselessly takes the scene of surrounding,
But during taking pictures, if being unsatisfied with to object space or wishing that certain objects not enter mirror in photo, user is logical at this time
Normal way is: it moves object and then returns to camera site from newly and take pictures, but when frequent progress is moved, or encounter body
When the heavier object of the bigger, weight ratio of product, does so and be undoubtedly a kind of predicament for photographer and worry, this problem can
To use image edit method provided by the invention to solve.
It is the concrete application flow chart of one image edit method of the embodiment of the present invention, the application flow packet referring to Fig. 7, Fig. 7
Include following steps:
Step 701: user opens the camera program/system realized based on image edit method of the invention.
Step 702: user clicks system and enters pose adjustment mode, can mark the institute identified in current scene at this time
There is example；
Step 703: user selects the instance objects to be adjusted (i.e. object instance object), and the specific adjustment based on user
Operational motion carries out the adjustment operation of step 704,705 or 706.
Step 704: if user thinks mobile object instance object, by mobile gesture, object instance object being moved to
Suitable position, while keeping object instance object original scale size constant, in moving process where object instance object
The deleted background of original area will be filled automatically.
Step 705: if user thinks that rotation is in rotation with the central point of object instance object to object instance object
The heart rotates object instance object, in rotary course, also carries out estimating building to the hidden area of target, realizes content
Auto-complete.
Step 706: if user wants to scale object instance object, with the central point of object instance object, to target reality
Example object zooms in or out operation.
704~step 706 of above-mentioned steps describes the movement to object instance object, rotation and zoom operations, operates
User can interact use in conjunction with three kinds of modes of operation in journey, such as shown in Fig. 8, when user is to sofa (object instance pair
As) carry out rotation process after, sofa position is constant, and angle changes；When user rotates sofa and after moving operation,
The position of sofa and angle are simultaneously displayed on different positions；After user moves and scales to sofa, sofa is with difference
Size be shown in different positions.
Step 707: after the completion of adjustment, user clicks camera function, shoots to edited scene.
Step 708: saving the photo of shooting, completion is this time taken pictures.
It is the process example of the adjustment to non-human object, the adjustment below in conjunction with Fig. 9 and Figure 10 to human object above
Process is illustrated.
In daily life, personage's camera shooting occupies the overwhelming majority in taking pictures, friend's out on tours together, university student's graduation
According to posing for photograph, relatives and friends' group photo is used all there may be posture is inconsistent or the unsatisfied situation of shooting result during taking pictures
Family is wanted to take to meet oneself conceives satisfied photo at heart, and the body posture and facial expression of photographic subjects generally require micro-
It adjusts.This problem can be used image edit method provided by the invention and solve.
It is the concrete application flow chart of two image edit method of the embodiment of the present invention, the application flow packet referring to Fig. 9, Fig. 9
Include following steps:
Step 901: user opens the camera program/system realized based on image edit method of the invention.
Step 902: the object instance object edited in user's click recognition photographed scene；
Step 903: system carries out example segmentation to object instance object, while establishing background model (i.e. to current background
Reject the image that object instance object obtains the background missing of object instance object occupied area), and automatic mapping object instance
Object establishes corresponding T-Pose model.
Step 904: user selects edit mode (mode one: to manually adjust；Mode two: Template Map).
Step 905: user's selection manually adjusts, and the movement of the adjusting based on user executes following several adjustment operations
One of or it is a variety of:
S9051, target position is relocated by mobile gesture；
S9052, size scaling is carried out to target by scaling gesture；
S9053, rotation process is carried out to target by rotation gesture, and fills deleted background automatically；
S9054, position is adjusted by clicking, pose adjustment is carried out to adjustment section position, during adjustment, in conjunction with kinematics original
It manages, the movement between each joint of linking.Therefore, need to maintain certain threshold value model to the adjustment operating result of human body different parts
In enclosing, such as arm number of rotation, arm moving distance etc. are based on human cinology's principle when moving to hand, need
Link arm position it is mobile.
Step 906: user selects Template Map, and system can provide body posture template and expression template and select for user at this time
(object instance object is selected state at this time) is selected, the template that can be selected based on user executes one of several adjustment operations
Or it is several:
S9061: user selects body posture template A, and system carries out body posture tune to object instance object according to template A
It is whole.
S9062, user select facial expression template B, and system carries out facial expression tune to object instance object according to template B
It is whole.
905~step 906 of above-mentioned steps describes manual mode to the movement of human object, rotation, scaling and body
Pose adjustment operation, in addition, also to Template Map mode to human object carry out body posture adjustment and facial expression adjust into
Introduction is gone.Such as shown in Figure 10, after the progress T-Pose mapping of object instance object, so that it may be carried out based on mapping result
The pose adjustment (including movement, rotation, scaling and body posture adjustment) of manual mode, can also carry out casting formwork mapping
The pose adjustment (including body posture adjustment and facial expression adjustment) of mode.
Step 907: after the completion of adjustment, user clicks camera function, shoots to edited scene.
Step 908: saving the photo of shooting, completion is this time taken pictures.
The implementation and application of image edit method of the embodiment of the present invention is described in detail above, the present invention also provides
A kind of image editing apparatus, is described in detail below in conjunction with Figure 11.
It is the structural schematic diagram of image editing apparatus of the embodiment of the present invention, as shown in figure 11, the device referring to Figure 11, Figure 11
Include:
Example cutting unit 1101, for all examples for including in image to be edited to be divided into example independent
Object；
Pose adjustment unit 1102, object instance object for being selected according to the request of the pose adjustment of user to user into
Row pose adjustment；
Edge amending unit 1103, for the adjustment edge adjusted of object instance object gesture in image to be edited into
The amendment of row edge.
It further include instance model unit 1104 in Figure 11 shown device；
The instance model unit 1104, for obtaining the example image of plurality of classes in advance and storing to example sample
Library is trained to obtain example network model to all example images in example sample database；
All examples for including in image to be edited are divided into example independent by the example cutting unit 1101
When object, be used for: the example network model trained in advance using instance model unit is to all realities for including in image to be edited
Example carries out identification and example segmentation, obtains the corresponding instance objects of different instances.
In Figure 11 shown device,
The instance model unit 1104 is trained to obtain example network to all example images in example sample database
When model, for executing following training operation to obtain the example network mould to each example image in example sample database
Type:
The characteristic pattern of the example image is extracted using convolutional neural networks CNN algorithm；
The object candidate area in the example image is extracted using region recommendation network RPN；
It is corrected pixel by ROIAlign algorithm, is aligned the pixel of object candidate area and characteristic pattern；
Divide and extract the region Mask for the instance objects for including in the example image based on full convolutional network FCN.
In Figure 11 shown device,
The pose adjustment includes mobile, scaling and rotation；
The pose adjustment unit 1102, object instance object user selected according to the request of the pose adjustment of user into
Row pose adjustment, comprising:
When the request of the pose adjustment of user is mobile object instance object, object instance object is moved to the appearance of user
The target position of state adjustment request instruction；
It is original with object instance object centers position when the request of the pose adjustment of user is scaling object instance object
Point zooms in and out operation to object instance object according to the scaling of the pose adjustment of user request instruction；
It is original with object instance object centers position when the request of the pose adjustment of user is rolling target instance objects
Point, using the horizontal direction of the plane of image to be edited, vertical direction, perpendicular to image to be edited direction as 3-D walls and floor
X-axis, y-axis and z-axis, according to the object instance object of the pose adjustment of user request instruction on x-axis, y-axis and z-axis direction
Angle is rotated, using 3D-GAN network model to the postrotational display on x-axis, y-axis and z-axis direction respectively of object instance object
Region carries out estimating building, and the building result of estimating according to object instance object in three dimensions of x-axis, y-axis and z-axis determines rotation
Display result of the object instance object in image to be edited after turning.
In Figure 11 shown device,
The object instance object is human object or non-human object.
In Figure 11 shown device,
When object instance object is human object,
The pose adjustment further comprises: body posture adjustment and facial expression adjustment；
The pose adjustment unit 1102, object instance object user selected according to the request of the pose adjustment of user into
Row pose adjustment, is further used for:
The corresponding T-Pose model of object instance object is established, and identifies the face key point of object instance object；
When the request of the pose adjustment of user is to carry out body posture adjustment to object instance object, if the posture of user
Adjustment request carry body posture Template Information, then using the T-Pose model by object instance object according to the morphological template into
The adjustment of row body posture；If the pose adjustment of user requests to carry adjustment location information and adjustment action message, utilizing should
T-Pose model adjust based on the body posture that the adjustment position and adjustment act to object instance object；
When the request of the pose adjustment of user is to carry out facial expression adjustment to object instance object, the posture of user is determined
The facial expression Template Information that adjustment request carries, using the T-Pose model by object instance object according to the facial expression mould
Plate carries out facial expression adjustment.
In Figure 11 shown device,
The pose adjustment unit 1102, object instance object user selected according to the request of the pose adjustment of user into
Before row pose adjustment, it is further used for: the object instance object that user selects is rejected from image to be edited, obtains target
The image to be edited of the occupied area background missing of instance objects；
The pose adjustment unit 1102, object instance object user selected according to the request of the pose adjustment of user into
After row pose adjustment, it is further used for: to the background absent region adjusted of object instance object gesture in image to be edited
Carry out background filling.
In Figure 11 shown device,
The pose adjustment unit 1102 lacks area to the background adjusted of object instance object gesture in image to be edited
When domain carries out background filling, it is used for:
Extract the global content characteristic figure of image to be edited；
Extract the Local textural feature figure on background absent region periphery；
According to the global content characteristic figure and Local textural feature figure, is lost based on confrontation and generate background absent region
Prognostic chart, and the prognostic chart is filled into background absent region.
From the above as can be seen that image edit method provided by the invention, can allow user more easily to edit
The targeted attitude of shooting and position, the mobile scaling and other effects of object in some photos of processing that can be more efficient can be with
Allow the adjustment human body attitude that user is more flexible, it is not necessary to which the posture of posing for photograph of regretting again is not in place.It thus can be very good to improve user's body
It tests, and editing process is not needed using specialized technical knowledge, picture editor's complexity can be greatly simplified.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the present invention.
Claims (16)
1. a kind of image edit method, which is characterized in that this method comprises:
All examples for including in image to be edited are divided into instance objects independent；
Pose adjustment is carried out to the object instance object that user selects according to the request of the pose adjustment of user；
Edge amendment is carried out to the adjustment edge adjusted of object instance object gesture in image to be edited.
2. the method according to claim 1, wherein
The example image of plurality of classes is obtained in advance and is stored and arrives example sample database, to all example images in example sample database
It is trained to obtain example network model；
The method that all examples for including in image to be edited are divided into instance objects independent are as follows: utilize training in advance
Example network model, identification is carried out to all examples for including in image to be edited and picture is carried out to each example for identifying
Plain grade segmentation, obtains the corresponding instance objects of each example.
3. according to the method described in claim 2, it is characterized in that,
The method for obtaining example network model is trained to all example images in example sample database are as follows:
To each example image in example sample database, following training operation is executed to obtain the example network model:
The characteristic pattern of the example image is extracted using convolutional neural networks CNN algorithm；
The object candidate area in the example image is extracted using region recommendation network RPN；
It is corrected pixel by ROIAlign algorithm, is aligned the pixel of object candidate area and characteristic pattern；
Divide and extract the region Mask for the instance objects for including in the example image based on full convolutional network FCN.
4. the method according to claim 1, wherein
The pose adjustment includes mobile, scaling and rotation；
Carrying out pose adjustment to the object instance object that user selects according to the request of the pose adjustment of user includes:
When the request of the pose adjustment of user is mobile object instance object, object instance object is moved to the posture tune of user
The target position of whole request instruction；
When the request of the pose adjustment of user is scaling object instance object, using object instance object centers position as origin, root
Operation is zoomed in and out to object instance object according to the scaling of the pose adjustment request instruction of user；
When the request of the pose adjustment of user is rolling target instance objects, using object instance object centers position as origin, with
The horizontal direction of the plane of image to be edited, vertical direction perpendicular to the direction of image to be edited are the x-axis of 3-D walls and floor, y
Axis and z-axis, according to rotation angle of the object instance object of the pose adjustment of user request instruction on x-axis, y-axis and z-axis direction
Degree, using 3D-GAN network model to object instance object respectively on x-axis, y-axis and z-axis direction postrotational display area into
Row estimates building, and according to object instance object three dimensions of x-axis, y-axis and z-axis estimate construct result determine it is postrotational
Display result of the object instance object in image to be edited.
5. according to the method described in claim 4, it is characterized in that,
The object instance object is human object or non-human object.
6. according to the method described in claim 5, it is characterized in that,
When object instance object is human object,
The pose adjustment further comprises: body posture adjustment and facial expression adjustment；
Pose adjustment is carried out to the object instance object that user selects according to the request of the pose adjustment of user, further comprises:
The corresponding T-Pose model of object instance object is established, and identifies the face key point of object instance object；
When the request of the pose adjustment of user is to carry out body posture adjustment to object instance object, if the pose adjustment of user
Request carries body posture Template Information, then object instance object is carried out body according to the morphological template using the T-Pose model
Body pose adjustment；If the pose adjustment request of user carries adjustment location information and adjustment action message, the T- is utilized
Pose model adjust based on the body posture that the adjustment position and adjustment act to object instance object；
When the request of the pose adjustment of user is to carry out facial expression adjustment to object instance object, the pose adjustment of user is determined
Request carry facial expression Template Information, using the T-Pose model by object instance object according to the facial expression template into
The adjustment of row facial expression.
7. according to method described in any claim of claim 4-6, which is characterized in that
Before carrying out pose adjustment to the object instance object that user selects according to the request of the pose adjustment of user, further wrap
It includes: the object instance object that user selects being rejected from image to be edited, obtains the occupied area background of object instance object
The image to be edited of missing；
After carrying out pose adjustment to the object instance object that user selects according to the request of the pose adjustment of user, further wrap
It includes: background filling is carried out to the background absent region adjusted of object instance object gesture in image to be edited.
8. the method according to the description of claim 7 is characterized in that
The method that background filling is carried out to the background absent region adjusted of object instance object gesture in image to be edited are as follows:
Extract the global content characteristic figure of image to be edited；
Extract the Local textural feature figure on background absent region periphery；
According to the global content characteristic figure and Local textural feature figure, the prediction for generating background absent region is lost based on confrontation
Figure, and the prognostic chart is filled into background absent region.
9. a kind of image editing apparatus, which is characterized in that the device includes:
Example cutting unit, for all examples for including in image to be edited to be divided into instance objects independent；
Pose adjustment unit, for carrying out posture tune to the object instance object that user selects according to the request of the pose adjustment of user
It is whole；
Edge amending unit is repaired for carrying out edge to the adjustment edge adjusted of object instance object gesture in image to be edited
Just.
10. device according to claim 9, which is characterized in that further include instance model unit；
The instance model unit, for obtaining the example image of plurality of classes in advance and storing to example sample database, to example
All example images in sample database are trained to obtain example network model；
The example cutting unit, when all examples for including in image to be edited are divided into instance objects independent,
For: the example network model trained in advance using instance model unit carries out all examples for including in image to be edited
It identifies and Pixel-level segmentation is carried out to each example identified, obtain the corresponding instance objects of each example.
11. device according to claim 10, which is characterized in that
The instance model unit, when being trained to obtain example network model to all example images in example sample database,
For executing following training operation to obtain the example network model to each example image in example sample database:
The characteristic pattern of the example image is extracted using convolutional neural networks CNN algorithm；
The object candidate area in the example image is extracted using region recommendation network RPN；
It is corrected pixel by ROIAlign algorithm, is aligned the pixel of object candidate area and characteristic pattern；
Divide and extract the region Mask for the instance objects for including in the example image based on full convolutional network FCN.
12. device according to claim 9, which is characterized in that
The pose adjustment includes mobile, scaling and rotation；
The pose adjustment unit carries out posture tune to the object instance object that user selects according to the request of the pose adjustment of user
It is whole, comprising:
When the request of the pose adjustment of user is mobile object instance object, object instance object is moved to the posture tune of user
The target position of whole request instruction；
When the request of the pose adjustment of user is scaling object instance object, using object instance object centers position as origin, root
Operation is zoomed in and out to object instance object according to the scaling of the pose adjustment request instruction of user；
When the request of the pose adjustment of user is rolling target instance objects, using object instance object centers position as origin, with
The horizontal direction of the plane of image to be edited, vertical direction perpendicular to the direction of image to be edited are the x-axis of 3-D walls and floor, y
Axis and z-axis, according to rotation angle of the object instance object of the pose adjustment of user request instruction on x-axis, y-axis and z-axis direction
Degree, using 3D-GAN network model to object instance object respectively on x-axis, y-axis and z-axis direction postrotational display area into
Row estimates building, and according to object instance object three dimensions of x-axis, y-axis and z-axis estimate construct result determine it is postrotational
Display result of the object instance object in image to be edited.
13. device according to claim 12, which is characterized in that
The object instance object is human object or non-human object.
14. device according to claim 13, which is characterized in that
When object instance object is human object,
The pose adjustment further comprises: body posture adjustment and facial expression adjustment；
The pose adjustment unit carries out posture tune to the object instance object that user selects according to the request of the pose adjustment of user
It is whole, further comprise:
The corresponding T-Pose model of object instance object is established, and identifies the face key point of object instance object；
When the request of the pose adjustment of user is to carry out body posture adjustment to object instance object, if the pose adjustment of user
Request carries body posture Template Information, then object instance object is carried out body according to the morphological template using the T-Pose model
Body pose adjustment；If the pose adjustment request of user carries adjustment location information and adjustment action message, the T- is utilized
Pose model adjust based on the body posture that the adjustment position and adjustment act to object instance object；
When the request of the pose adjustment of user is to carry out facial expression adjustment to object instance object, the pose adjustment of user is determined
Request carry facial expression Template Information, using the T-Pose model by object instance object according to the facial expression template into
The adjustment of row facial expression.
15. device described in any claim of 2-14 according to claim 1, which is characterized in that
The pose adjustment unit carries out posture tune to the object instance object that user selects according to the request of the pose adjustment of user
Before whole, it is further used for: the object instance object that user selects is rejected from image to be edited, obtains object instance object
Occupied area background missing image to be edited；
The pose adjustment unit carries out posture tune to the object instance object that user selects according to the request of the pose adjustment of user
After whole, it is further used for: background is carried out to the background absent region adjusted of object instance object gesture in image to be edited
Filling.
16. device according to claim 15, which is characterized in that
The pose adjustment unit carries on the back the background absent region adjusted of object instance object gesture in image to be edited
When scape is filled, it is used for:
Extract the global content characteristic figure of image to be edited；
Extract the Local textural feature figure on background absent region periphery；
According to the global content characteristic figure and Local textural feature figure, the prediction for generating background absent region is lost based on confrontation
Figure, and the prognostic chart is filled into background absent region.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811265226.9A CN109472795A (en) | 2018-10-29 | 2018-10-29 | A kind of image edit method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811265226.9A CN109472795A (en) | 2018-10-29 | 2018-10-29 | A kind of image edit method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109472795A true CN109472795A (en) | 2019-03-15 |
Family
ID=65666524
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811265226.9A Pending CN109472795A (en) | 2018-10-29 | 2018-10-29 | A kind of image edit method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109472795A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109618097A (en) * | 2018-12-29 | 2019-04-12 | 维沃移动通信有限公司 | Auxiliary photo-taking method and terminal device |
| CN111246113A (en) * | 2020-03-05 | 2020-06-05 | 上海瑾盛通信科技有限公司 | Image processing method, device, equipment and storage medium |
| CN111488778A (en) * | 2019-05-29 | 2020-08-04 | 北京京东尚科信息技术有限公司 | Image processing method and apparatus, computer system, and readable storage medium |
| CN111784799A (en) * | 2020-06-30 | 2020-10-16 | 北京百度网讯科技有限公司 | Image filling method, device, equipment and storage medium |
| CN111800574A (en) * | 2020-06-23 | 2020-10-20 | 维沃移动通信有限公司 | Imaging method and device and electronic equipment |
| CN113012042A (en) * | 2019-12-20 | 2021-06-22 | 海信集团有限公司 | Display device, virtual photo generation method, and storage medium |
| CN113706723A (en) * | 2021-08-23 | 2021-11-26 | 维沃移动通信有限公司 | Image processing method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106228590A (en) * | 2016-07-19 | 2016-12-14 | 中国电子科技集团公司第二十八研究所 | A kind of human body attitude edit methods in image |
| CN107507126A (en) * | 2017-07-27 | 2017-12-22 | 大连和创懒人科技有限公司 | A kind of method that 3D scenes are reduced using RGB image |
| CN107705365A (en) * | 2017-09-08 | 2018-02-16 | 郭睿 | Editable three-dimensional (3 D) manikin creation method, device, electronic equipment and computer program product |
| CN108701352A (en) * | 2016-03-23 | 2018-10-23 | 英特尔公司 | Amending image using the identification based on three dimensional object model and enhancing |
-
2018
- 2018-10-29 CN CN201811265226.9A patent/CN109472795A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108701352A (en) * | 2016-03-23 | 2018-10-23 | 英特尔公司 | Amending image using the identification based on three dimensional object model and enhancing |
| CN106228590A (en) * | 2016-07-19 | 2016-12-14 | 中国电子科技集团公司第二十八研究所 | A kind of human body attitude edit methods in image |
| CN107507126A (en) * | 2017-07-27 | 2017-12-22 | 大连和创懒人科技有限公司 | A kind of method that 3D scenes are reduced using RGB image |
| CN107705365A (en) * | 2017-09-08 | 2018-02-16 | 郭睿 | Editable three-dimensional (3 D) manikin creation method, device, electronic equipment and computer program product |
Non-Patent Citations (3)
| Title |
|---|
| CHAO YANG等: "High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis", 《HTTPS://ARXIV.ORG/ABS/1611.09969》 * |
| JIAJUN WU等: "Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling", 《HTTPS://ARXIV.ORG/ABS/1610.07584》 * |
| KAIMING HE等: "Mask R-CNN", 《HTTPS://ARXIV.ORG/ABS/1703.06870》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109618097A (en) * | 2018-12-29 | 2019-04-12 | 维沃移动通信有限公司 | Auxiliary photo-taking method and terminal device |
| CN109618097B (en) * | 2018-12-29 | 2021-03-16 | 维沃移动通信有限公司 | Auxiliary photographing method and terminal equipment |
| CN111488778A (en) * | 2019-05-29 | 2020-08-04 | 北京京东尚科信息技术有限公司 | Image processing method and apparatus, computer system, and readable storage medium |
| CN113012042A (en) * | 2019-12-20 | 2021-06-22 | 海信集团有限公司 | Display device, virtual photo generation method, and storage medium |
| CN113012042B (en) * | 2019-12-20 | 2023-01-20 | 海信集团有限公司 | Display device, virtual photo generation method, and storage medium |
| CN111246113A (en) * | 2020-03-05 | 2020-06-05 | 上海瑾盛通信科技有限公司 | Image processing method, device, equipment and storage medium |
| CN111800574A (en) * | 2020-06-23 | 2020-10-20 | 维沃移动通信有限公司 | Imaging method and device and electronic equipment |
| CN111800574B (en) * | 2020-06-23 | 2022-06-24 | 维沃移动通信有限公司 | Imaging method and device and electronic equipment |
| CN111784799A (en) * | 2020-06-30 | 2020-10-16 | 北京百度网讯科技有限公司 | Image filling method, device, equipment and storage medium |
| CN111784799B (en) * | 2020-06-30 | 2024-01-12 | 北京百度网讯科技有限公司 | Image filling method, device, equipment and storage medium |
| CN113706723A (en) * | 2021-08-23 | 2021-11-26 | 维沃移动通信有限公司 | Image processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109472795A (en) | A kind of image edit method and device | |
| JP7176012B2 (en) | OBJECT MODELING OPERATING METHOD AND APPARATUS AND DEVICE | |
| CN110515452B (en) | Image processing method, image processing device, storage medium and computer equipment | |
| US10679046B1 (en) | Machine learning systems and methods of estimating body shape from images | |
| CN108230240B (en) | Method for obtaining position and posture in image city range based on deep learning | |
| WO2021174939A1 (en) | Facial image acquisition method and system | |
| CN114981844A (en) | 3D body model generation | |
| JP5244951B2 (en) | Apparatus and system for image processing based on 3D spatial dimensions | |
| WO2015188684A1 (en) | Three-dimensional model reconstruction method and system | |
| CN107590708B (en) | Method and device for generating user specific body shape model | |
| US11836866B2 (en) | Deforming real-world object using an external mesh | |
| CN111652123B (en) | Image processing and image synthesizing method, device and storage medium | |
| CN113628327A (en) | Head three-dimensional reconstruction method and equipment | |
| US11688136B2 (en) | 3D object model reconstruction from 2D images | |
| CN109509241A (en) | Based on the bone reorientation method of quaternary number in role animation | |
| US20230154129A1 (en) | Body normal network light and rendering control | |
| US20230267687A1 (en) | 3d object model reconstruction from 2d images | |
| WO2020134925A1 (en) | Illumination detection method and apparatus for facial image, and device and storage medium | |
| WO2024088071A1 (en) | Three-dimensional scene reconstruction method and apparatus, device and storage medium | |
| CN116917938A (en) | Visual effect of whole body | |
| Reinert et al. | Animated 3D creatures from single-view video by skeletal sketching. | |
| CN117136381A (en) | whole body segmentation | |
| US11430168B2 (en) | Method and apparatus for rigging 3D scanned human models | |
| JPH08305894A (en) | Three-dimensional image generating device capable of representing wrinkle | |
| CN113554745B (en) | Three-dimensional face reconstruction method based on image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190315 |
|
| RJ01 | Rejection of invention patent application after publication |