CN117011417A

CN117011417A - Image processing method and device and computer equipment

Info

Publication number: CN117011417A
Application number: CN202310802827.3A
Authority: CN
Inventors: 张昕荻; 卓力安; 王琪; 张邦; 田林睿; 孙可; 曹健
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2023-06-30
Filing date: 2023-06-30
Publication date: 2023-11-07

Abstract

The application discloses an image processing method, an image processing device and computer equipment. Wherein the method comprises the following steps: acquiring a first image, wherein a first object in the first image is worn with clothing; performing clothing segmentation on the first image to obtain a clothing mask of clothing; obtaining object generation parameters; and inputting the first image into a diffusion generation model, shielding the clothes by adopting a clothes mask, and generating a second object based on the object generation parameters to obtain a second image. The application solves the technical problem that the fitting degree of clothes and objects in the display images is low when the display images are generated in the related technology.

Description

Image processing method and device and computer equipment

Technical Field

The present application relates to the field of image processing, and in particular, to an image processing method, an image processing device, and a computer device.

Background

In the related art, when a display image of a garment worn by a subject needs to be generated, the garment image is generally directly superimposed on the subject image, and the generated display image has the technical problem that the fitting degree of the garment and the subject is low.

In view of the above problems, no effective solution has been proposed at present.

Disclosure of Invention

The embodiment of the application provides an image processing method, an image processing device and computer equipment, which at least solve the technical problem that the fitting degree of clothes and objects in display images is low when the display images are generated in the related technology.

According to an aspect of an embodiment of the present application, there is provided an image processing method including: acquiring a first image, wherein a first object in the first image is worn with clothes; performing clothing segmentation on the first image to obtain a clothing mask of the clothing; obtaining object generation parameters; and inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters to obtain a second image.

According to an aspect of an embodiment of the present application, there is provided an image processing method including: responding to an image input instruction acted on an operation interface, and displaying a first image on the operation interface, wherein a first object in the first image is worn with clothes; responding to a clothing segmentation instruction acting on the operation interface, and displaying a clothing mask of the clothing on the operation interface, wherein the clothing mask is obtained by performing clothing segmentation on the first image; responding to a parameter input instruction acted on the operation interface, and displaying input object generation parameters on the operation interface; and responding to an image generation instruction acted on the operation interface, and displaying a second image on the operation interface, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameter.

According to an aspect of an embodiment of the present application, there is provided an image processing method including: displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR equipment or the AR equipment to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters.

According to an aspect of an embodiment of the present application, there is provided an image processing method including: acquiring a first image by calling a first interface, wherein a first object in the first image is worn with clothes; outputting a clothing mask of the clothing by calling a second interface, wherein the clothing mask of the clothing is obtained by carrying out clothing segmentation on the first image; and acquiring object generation parameters by calling a third interface, and generating a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters.

According to an aspect of an embodiment of the present application, there is provided an image processing apparatus including: the first acquisition module is used for acquiring a first image, wherein a first object in the first image is worn with clothes; the segmentation module is used for performing clothing segmentation on the first image to obtain a clothing mask of the clothing; the second acquisition module is used for acquiring object generation parameters; and the processing module is used for inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters to obtain a second image.

According to an aspect of an embodiment of the present application, there is provided an image processing apparatus including: the first display module is used for responding to an image input instruction acted on an operation interface, and displaying a first image on the operation interface, wherein a first object in the first image is worn with clothes; the second display module is used for responding to a clothing segmentation instruction acting on the operation interface and displaying a clothing mask of the clothing on the operation interface, wherein the clothing mask is obtained by carrying out clothing segmentation on the first image; the third display module is used for responding to a parameter input instruction acted on the operation interface and displaying input object generation parameters on the operation interface; and the fourth display module is used for responding to an image generation instruction acted on the operation interface, and displaying a second image on the operation interface, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameter.

According to an aspect of an embodiment of the present application, there is provided an image processing apparatus including: the display module is used for displaying a first image on a display picture of the virtual reality VR equipment or the augmented reality AR equipment, wherein a first object in the first image is worn with clothes; the receiving module is used for receiving the image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; the selection module is used for selecting object generation parameters; the driving module is used for driving the VR equipment or the AR equipment to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters.

According to an aspect of an embodiment of the present application, there is provided an image processing apparatus including: the first calling module is used for obtaining a first image by calling a first interface, wherein a first object in the first image is worn with clothes; the second calling module is used for outputting a clothing mask of the clothing by calling a second interface, wherein the clothing mask of the clothing is obtained by carrying out clothing segmentation on the first image; and the third calling module is used for obtaining object generation parameters by calling a third interface and generating a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters.

According to an aspect of an embodiment of the present application, there is provided a computer apparatus including: a memory storing an executable program; and a processor for running the program, wherein the program executes the image processing method according to any one of the above.

According to an aspect of an embodiment of the present application, there is provided a computer-readable storage medium including a stored executable program, wherein the executable program, when executed, controls a device in which the computer-readable storage medium is located to perform the image processing method of any one of the above.

In the embodiment of the application, a first image comprising a first object wearing clothes is obtained, clothes segmentation is carried out on the first image, a clothes mask of the clothes is obtained, and then object generation parameters are obtained. And inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters, so that the aim of obtaining the second image is finally achieved. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, and the generated second object is still in a natural state that the object wears the original clothing. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

fig. 1 is a schematic diagram of a hardware environment of a virtual reality device of an image processing method according to an embodiment of the application;

FIG. 2 is a block diagram of a computing environment for an image processing method according to an embodiment of the present application;

fig. 3 is a flowchart of an image processing method one according to embodiment 1 of the present application;

fig. 4 is a flowchart of an image processing method two according to embodiment 1 of the present application;

fig. 5 is a flowchart of an image processing method three according to embodiment 1 of the present application;

fig. 6 is a flowchart of an image processing method four according to embodiment 1 of the present application;

FIG. 7 is a schematic diagram of an image processing method provided in accordance with an alternative embodiment of the present application;

fig. 8 is a block diagram of the structure of an image processing apparatus one according to an embodiment of the present application;

Fig. 9 is a block diagram of the structure of an image processing apparatus two according to the embodiment of the present application;

fig. 10 is a block diagram of the structure of an image processing apparatus three according to an embodiment of the present application;

fig. 11 is a block diagram of the structure of an image processing apparatus four according to an embodiment of the present application;

fig. 12 is a block diagram of a computer terminal according to an embodiment of the present application.

Detailed Description

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

First, partial terms or terminology appearing in the course of describing embodiments of the application are applicable to the following explanation:

AIGC: AI generated content, also called as generation type AI, means artificial intelligence generation content, and the scheme particularly refers to the generation of a clothing try-on effect diagram based on a diffusion model.

SD: stable Diffusion algorithm is an image Stable Diffusion method capable of smoothing images and maintaining edge information, and comprises a generated large network model structure. It reduces noise by adding diffusion terms to the image while preserving detail and edge information of the image. The stable diffusion method uses a diffusion coefficient to control the smoothness of an image. A smaller diffusion coefficient will allow the image to retain more detail and edge information, but may result in increased noise. A larger diffusion coefficient will more strongly smooth the image, reducing noise. The stable diffusion method is generally implemented by iteratively applying a diffusion operation. Each pixel in the image is updated according to the intensity of its neighboring pixels for each iteration. In this way, each pixel in the image is affected by its surrounding pixels, thereby achieving a smoothing effect. The stable diffusion method can be applied to image processing tasks such as image denoising, edge detection and the like. The method can effectively smooth the image, maintain the detail and edge information of the image and improve the image quality.

Control Net: a model generated by a fine control SD.

Diffuse: diffusion algorithms, an algorithm or technique for smoothing images in image processing. It reduces noise and detail of the image by propagating differences between pixels in the image. The diffusion algorithm averages pixel values around them based on gradient information between pixels, thereby reducing local variations in the image. This smoothing process may make the image appear more blurred, but may remove noise and detail, thereby improving the quality of the image.

Warp: refers to an operation of deforming or deforming an image. Through Warping, translation, rotation, scaling, warping, etc. of the image can be achieved, thereby changing the shape or perspective relationship of the image.

Example 1

According to an embodiment of the present application, there is provided an image processing method, it being noted that the steps shown in the flowcharts of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.

Fig. 1 is a schematic diagram of a hardware environment of a virtual reality device of an image processing method according to an embodiment of the application. As shown in fig. 1, the virtual reality device 104 is connected to the terminal 106, the terminal 106 is connected to the server 102 via a network, and the virtual reality device 104 is not limited to: the terminal 104 is not limited to a PC, a mobile phone, a tablet computer, etc., and the server 102 may be a server corresponding to a media file operator, and the network includes, but is not limited to: a wide area network, a metropolitan area network, or a local area network.

Optionally, the virtual reality device 104 of this embodiment includes: memory, processor, and transmission means. The memory is used to store an application program that can be used to perform: acquiring a first image, wherein a first object in the first image is worn with clothing; performing clothing segmentation on the first image to obtain a clothing mask of clothing; obtaining object generation parameters; and inputting the first image into a diffusion generation model, shielding the clothes by adopting a clothes mask, and generating a second object based on the object generation parameters to obtain a second image. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, the generated second object is still the natural state of the original clothing, and the clothing still has the real physical effects of folds, pendants and the like. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

The terminal of this embodiment may be configured to perform displaying a first image on a presentation screen of a Virtual Reality (VR) device or an augmented Reality (Augmented Reality AR) device, where a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR device or the AR device to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

Optionally, the HMD (Head MountDisplay, head mounted display) head display and the eye tracking module of the virtual reality device 104 of this embodiment have the same functions as those of the above embodiment, that is, a screen in the HMD head display is used for displaying a real-time picture, and the eye tracking module in the HMD is used for acquiring a real-time motion track of an eyeball of a user. The terminal of the embodiment obtains the position information and the motion information of the user in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the head of the user in the virtual three-dimensional space and the visual field orientation of the user in the virtual three-dimensional space.

The hardware architecture block diagram shown in fig. 1 may be used not only as an exemplary block diagram for an AR/VR device (or mobile device) as described above, but also as an exemplary block diagram for a server as described above, and in an alternative embodiment, fig. 2 shows in block diagram form one embodiment of a computing node in a computing environment 201 using an AR/VR device (or mobile device) as described above in fig. 1. Fig. 2 is a block diagram of a computing environment for an image processing method according to an embodiment of the present application, as shown in fig. 2, the computing environment 201 includes a plurality of computing nodes (e.g., servers) running on a distributed network (shown as 210-1, 210-2, …). Different computing nodes contain local processing and memory resources and end user 202 may run applications or store data remotely in computing environment 201. The application may be provided as a plurality of services 220-1, 220-2, 220-3, and 220-4 in computing environment 201, representing services "A", "D", "E", and "H", respectively.

End user 202 may provide and access services through a web browser or other software application on a client, in some embodiments, provisioning and/or requests of end user 202 may be provided to portal gateway 230. Ingress gateway 230 may include a corresponding agent to handle provisioning and/or request for services (one or more services provided in computing environment 201).

Services are provided or deployed in accordance with various virtualization techniques supported by the computing environment 201. In some embodiments, services may be provided according to Virtual Machine (VM) based virtualization, container based virtualization, and/or the like. Virtual machine-based virtualization may be the emulation of a real computer by initializing a virtual machine, executing programs and applications without directly touching any real hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container may be started to virtualize the entire Operating System (OS) so that multiple workloads may run on a single Operating System instance.

In one embodiment based on container virtualization, several containers of a service may be assembled into one Pod (e.g., kubernetes Pod). For example, as shown in FIG. 2, the service 220-2 may be equipped with one or more Pods 240-1, 240-2, …,240-N (collectively referred to as Pods). The Pod may include an agent 245 and one or more containers 242-1, 242-2, …,242-M (collectively referred to as containers). One or more containers in the Pod handle requests related to one or more corresponding functions of the service, and the agent 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services may also be Pod similar to Pod.

In operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. As shown in FIG. 2, service "A"220-1 receives a user request of end user 202 from ingress gateway 230, service "A"220-1 may invoke service "D"220-2, and service "D"220-2 may request service "E"220-3 to perform one or more functions.

The computing environment may be a cloud computing environment, and the allocation of resources is managed by a cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of the server. The computing environment allows developers to execute code that responds to events without building or maintaining a complex infrastructure. Instead of expanding a single hardware device to handle the potential load, the service may be partitioned to a set of functions that can be automatically scaled independently.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

In the above-described operating environment, the present application provides an image processing method as shown in fig. 3. Fig. 3 is a flowchart of an image processing method one according to embodiment 1 of the present application, as shown in fig. 3, the method includes the steps of:

step S302, a first image is obtained, wherein a first object in the first image is worn with clothes;

in step S302 provided in the present application, the first image may be any image of the first object wearing the garment, for example, may be any image of the object, may be any image of the garment, and may be any image of the gesture.

The first object may include any object that may be used to wear clothing, such as a person, an animal, etc., and the first object may be set in a wide range, such as a personified person, a cartoon person, a real person, etc., without limitation.

Wherein the apparel is an apparel that the first object may wear, e.g., an apparel that may be worn on the body, hand-held appliances, head-wearable accessories, and the like, without limitation.

As an alternative embodiment, the execution subject of the method of the present embodiment may be a terminal or a server for processing an image, for example, when applied to a terminal for executing an image processing, the image processing in a simple image scene can be easily implemented; for example, when the method is applied to a server, abundant computing resources of the server can be called, or a relatively huge and accurate image processing model can be used for processing images more accurately.

The types of the terminals may be various, for example, a mobile terminal having a certain computing capability, a fixed computer device having an identification capability, or the like. The types of the servers may be various, for example, a local server or a virtual cloud server. The server may be a single computer device according to its computing power, or may be a computer cluster in which a plurality of computer devices are integrated.

Step S304, performing clothing segmentation on the first image to obtain a clothing mask of the clothing;

in step S304 provided by the present application, the first image is subjected to clothing segmentation, and the clothing part in the first image is segmented and processed to obtain the clothing mask of the clothing. A apparel mask is a binary image having the same size as an image of an apparel portion, where a pixel value of 1 indicates that the pixel should be preserved during processing, and a pixel value of 0 indicates that the pixel should be ignored or occluded during processing. I.e. in a subsequent operation, the masking operation of a specific region in the image may be achieved by masking. The region with the pixel value of 1 in the mask can be reserved, other regions can be set to 0 or replaced by other values, namely, the other regions can be transformed, and the purpose of keeping the region with the pixel value of 1, namely, the clothing part, unchanged is achieved.

Step S306, obtaining object generation parameters;

in step S306 provided in the present application, the object generation parameters are parameters required when the first object is generated as the second object. The parameter may be generated in a variety of ways. Such as may be generated from keywords, controls selected by the user, or any other manner of generation.

The parameter may relate to various aspects of generating parameters, such as a change in the hair style of the second object as compared to the first object, where the object generating parameter may be a hair style related parameter, such as a long hair and a short hair related parameter, and changing the hair of the first object to obtain the second object. For example, the second object may have a skin color and a facial feature that are different from those of the first object, and the object generating parameter may be a parameter related to the skin color and the facial feature, such as a wheat skin color and a parameter related to the delicate facial feature, and the skin color and the facial feature of the first object are changed to obtain the second object. For example, the second image may have a background that is changed compared to the first image, where the object generating parameter may be an environmental background where the second object is located, for example, the background in the second image is a forest, where the object generating parameter may be a parameter related to the forest background, and the background in the first image is changed to obtain the second image. Specific object generation parameters are not limited herein, and may be set in a customized manner according to actual applications and scenes.

Step S308, inputting the first image into a diffusion generation model, shielding the clothing by using a clothing mask, and generating a second object based on the object generation parameters to obtain a second image.

In step S308 provided by the present application, the first image is input into the diffusion generation model so as to generate the second image based on the first image. In the process, the clothing mask is adopted to shield the clothing, so that the clothing part is unchanged, namely, the real physical effects of folds, pendants and the like of the clothing when the clothing is worn on the object are reserved. And replacing the first object with the second image based on the mode of generating the second object by the object generation parameters to obtain a second image of wearing apparel of the second object.

The diffusion generation model may be a diffusion model, an SD model, or the like, and may be adaptively selected according to actual applications and scenes.

Through the steps, a first image comprising the first object wearing the clothing is obtained, the first image is subjected to clothing segmentation to obtain a clothing mask of the clothing, and then the object generation parameters are obtained. And inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters, so that the aim of obtaining the second image is finally achieved. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, the generated second object is still the natural state of the original clothing, and the clothing still has the real physical effects of folds, pendants and the like. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

As an alternative embodiment, acquiring the first image includes: acquiring an original image of a first object wearing apparel; and segmenting the first object and the clothes from the original image, and overlapping the segmented first object and the segmented clothes into the target background image to obtain a first image.

In this embodiment, an original image of a first object wearing apparel is obtained, the original image may be an original image of any background, the first object and apparel are segmented from the original image, the purpose of separating the first object and apparel from the background in the original image can be achieved, then the segmented first object and apparel are superimposed into the target background image, a first image of the first object and apparel under the target background can be obtained, the first object and apparel under the specified target background can be obtained, and a second image of the specified background can be finally obtained.

And the first object and the clothes are segmented from the original image and are overlapped in the target background image to obtain the first image, so that the background can be more attached, and the success rate and harmony of changing the background are improved.

The target background can be a solid background so that a diffusion generation model can be better used, and the obtained second image meets requirements.

As an alternative embodiment, the object generation parameters are obtained, including at least one of: acquiring object attribute parameters of a second object to be generated; and obtaining object environment parameters of the second object to be generated.

In this embodiment, the object attribute parameter and the object environment parameter of the second object are referred to, wherein the object attribute parameter may be a parameter related to the object wearing the apparel itself, and the object environment parameter may be a parameter related to a background portion other than the first object and the apparel in the first image, or a parameter related to a lighting aspect of the first image. Object attribute parameters such as skin color, facial features, hairstyle, etc., may change the first object to obtain the second object. The object environment parameter may be a background parameter, the light parameter, for example, the background is sea, forest, etc., the optical fiber is dim light, white light, etc., and the background in the first image may be changed to obtain the background in the second image. Thus, by the two parameters, the part except for the clothes in the first image can be changed, and the changed second image can be obtained. The aim of obtaining the clothing display diagram with the real physical effects of wrinkles, pendants and the like is achieved. And the generated second image can be obtained by converting the first image into any parameter, so that the requirements of users can be met, and images which better meet the preferences of the users can be manufactured.

As an alternative embodiment, the method further comprises: performing edge detection on the clothing mask to obtain a clothing outline; inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image, wherein the method comprises the following steps: inputting the first image into a diffusion generation model, taking the outline of the clothes as edge constraint of the clothes, shielding the clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image.

In this embodiment, edge detection is performed on the clothing mask, so as to determine the outline of the clothing, which is used to constrain the clothing, so as to avoid the constraint that the clothing exceeds the clothing outline in the generated second image, that is, by setting the clothing outline, the algorithm can be restricted to operate in the area outside the clothing outline, and the clothing in the second image can be ensured to be unchanged. By setting the clothing outline, the image processing effect can be improved, and by using the clothing outline to carry out constraint, the algorithm can be limited to operate outside the clothing area, so that the accuracy and effect of the algorithm are improved, and the clothing part in the generated second image is unchanged.

As an alternative embodiment, the method further comprises: obtaining an object contour of a first object based on the first image; inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image, wherein the method comprises the following steps: inputting the first image into a diffusion generation model, taking the outline of the object as the edge constraint of the second object, shielding the clothing by adopting a clothing mask, and generating the second object based on the object generation parameters to obtain a second image.

In this embodiment, the first object is processed, and an object contour of the first object may be obtained by an algorithm such as edge detection. The constraint is used for constraining the object, and the constraint that the second object in the generated second image exceeds the object outline is avoided, namely, by setting the object outline, the operation of an area in the object outline when the algorithm generates the second object can be limited when the algorithm generates the second object, and the second object similar to the body height of the first object can be generated. By setting the object contour, the image processing effect can be improved, and by using the object contour to carry out constraint, the algorithm can be limited to operate within the limitation of the object region when the second object is generated, so that the accuracy and the effect of the algorithm are improved. When a second object similar to the first object in height and shape is worn, the effect of displaying the clothing with the real physical effects such as wrinkles, sagging and the like can be achieved. And because the object contour is used for constraint, the generated limbs, trunk and the like of the second object can be more reasonable, and the phenomenon of generating abnormal objects is avoided.

As an alternative embodiment, obtaining the object contour of the first object based on the first image includes: converting the first image into a solid background image in case the first image is colored; and carrying out edge detection on the first object in the solid background image to obtain the object outline of the first object.

In this embodiment, the steps in determining the contour of an object are described. The first image is converted into the solid background image, so that edge detection can be facilitated, interference information can be reduced, and the detected object outline is more accurate. Moreover, by converting the first image into a solid background image, the interference of objects in the background can be eliminated due to the simple background, the extraction of contours in the background can be avoided, and the representation and processing of the image can be greatly simplified due to the simple background, so that the complexity of storage and calculation can be reduced. Moreover, by converting the first image into a solid color, some specific features in the image, such as edges, etc., can also be extracted, i.e. extraction of the object contour can be facilitated by this process.

As an alternative embodiment, the first object and the second object are objects with different images and/or background environments, and the objects are fit models.

In this embodiment, the object is a fit model, which can achieve the purpose of displaying the clothing. When different second images are generated through the first images, the purpose of generating different clothing display diagrams can be achieved, the different clothing display diagrams can be as described above, the images of objects in the diagrams are different, or the backgrounds are different, custom setting can be carried out according to actual application and scenes, and the method can cater to the preferences of different users. In addition, when the second object in the obtained second image wears the clothes, the clothes can also have the real physical effects of wrinkles, sagging and the like. Therefore, the generated second image not only accords with the user preference, but also truly shows the state that the clothes are worn on the object. Moreover, can be for the clothing display diagram that the clothing is comparable to professional photography, can produce a large amount of clothing display diagrams fast, can be different models in a large amount of clothing display diagrams, different backgrounds, very convenient and image quality is very good, can improve the show quality greatly when guaranteeing low cost.

Fig. 4 is a flowchart of a second image processing method according to embodiment 1 of the present invention, as shown in fig. 4, the method includes the steps of:

Step S402, responding to an image input instruction acted on an operation interface, and displaying a first image on the operation interface, wherein a first object in the first image is worn with clothes;

step S404, responding to a clothing segmentation instruction acted on an operation interface, and displaying a clothing mask of the clothing on the operation interface, wherein the clothing mask is obtained by performing clothing segmentation on the first image;

step S406, in response to the parameter input instruction acting on the operation interface, displaying the input object generation parameter on the operation interface;

in step S408, in response to the image generation instruction acting on the operation interface, a second image is displayed on the operation interface, where the second image is obtained by inputting the first image into the diffusion generation model, shielding the clothing with a clothing mask, and generating a second object based on the object generation parameter.

In this step, a first image including a first object wearing apparel is displayed on an operation interface in response to an image input instruction acting on the operation interface, an apparel mask of apparel obtained by apparel division of the first image is displayed on the operation interface in response to an apparel division instruction acting on the operation interface, input object generation parameters are displayed on the operation interface in response to a parameter input instruction acting on the operation interface, and then a second image obtained by diffusion generation model of the first image input is displayed on the operation interface in response to an image generation instruction acting on the operation interface, apparel is blocked by the apparel mask, and a second object is generated based on the object generation parameters. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, the generated second object is still the natural state of the original clothing, and the clothing still has the real physical effects of folds, pendants and the like. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

There is also provided, in accordance with an embodiment of the present application, an image processing method applicable to virtual reality scenes such as virtual reality VR devices, augmented reality AR devices, etc., it being noted that the steps shown in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in an order other than that shown herein.

Fig. 5 is a flowchart of an image processing method three according to embodiment 1 of the present application, as shown in fig. 5, the method including the steps of:

step S502, a first image is displayed on a display screen of a virtual reality VR device or an augmented reality AR device, wherein a first object in the first image is worn with clothes;

step S504, receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing;

step S506, selecting object generation parameters;

in step S508, the VR device or the AR device is driven to display a second image, where the second image is obtained by inputting the first image into the diffusion generation model, masking the clothing with a clothing mask, and generating a second object based on the object generation parameter.

In this step, a first image comprising a first object wearing apparel is presented on a presentation screen of a virtual reality VR device or an augmented reality AR device. And receiving an image processing instruction, performing clothing segmentation on the first image to obtain a clothing mask of the clothing, selecting object generation parameters to drive VR equipment or AR equipment to display a second image obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, the generated second object is still the natural state of the original clothing, and the clothing still has the real physical effects of folds, pendants and the like. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

Alternatively, in the present embodiment, the above-described image processing method may be applied to a hardware environment constituted by a server, a virtual reality device. The second image is shown on a presentation screen of the virtual reality VR device or the augmented reality AR device, and the server may be a server corresponding to a media file operator, where the network includes but is not limited to: the virtual reality device is not limited to a wide area network, a metropolitan area network, or a local area network: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, and the like.

Optionally, the virtual reality device comprises: memory, processor, and transmission means. The memory is used to store an application program that can be used to perform: displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR device or the AR device to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

It should be noted that, the above-mentioned image processing method applied to the VR device or the AR device in this embodiment may include the method of the embodiment shown in fig. 5, so as to achieve the purpose of driving the VR device or the AR device to display the second image.

Alternatively, the processor of this embodiment may call the application program stored in the memory through the transmission device to perform the above steps. The transmission device can receive the media file sent by the server through the network and can also be used for data transmission between the processor and the memory.

Optionally, in the virtual reality device, a head-mounted display with eye tracking is provided, a screen in the head-mounted display of the HMD is used for displaying a video picture displayed, an eye tracking module in the HMD is used for acquiring real-time motion tracks of eyes of the user, a tracking system is used for tracking position information and motion information of the user in a real three-dimensional space, a calculation processing unit is used for acquiring real-time position and motion information of the user from the tracking system, and calculating three-dimensional coordinates of the head of the user in the virtual three-dimensional space, visual field orientation of the user in the virtual three-dimensional space and the like.

In the embodiment of the present application, the virtual reality device may be connected to a terminal, where the terminal and the server are connected through a network, and the virtual reality device is not limited to: the terminal is not limited to a PC, a mobile phone, a tablet PC, etc., and the server may be a server corresponding to a media file operator, and the network includes but is not limited to: a wide area network, a metropolitan area network, or a local area network.

Fig. 6 is a flowchart of an image processing method four according to embodiment 1 of the present invention, as shown in fig. 6, the method including the steps of:

step S602, a first image is obtained by calling a first interface, wherein a first object in the first image is worn with clothes;

step S604, outputting a clothing mask of the clothing by calling a second interface, wherein the clothing mask of the clothing is obtained by carrying out clothing segmentation on the first image;

step S606, obtaining object generation parameters by calling a third interface, and generating a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by using a clothing mask, and generating a second object based on the object generation parameters.

Through the steps, a first image comprising a first object wearing clothes is obtained through calling a first interface, a clothes mask of the clothes obtained by cutting the first image into clothes is output through calling a second interface, object generation parameters are obtained through calling a third interface, a diffusion generation model is generated by inputting the first image into the diffusion generation model, the clothes mask is adopted to shade the clothes, and a second image obtained through generating a second object based on the object generation parameters is obtained. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, the generated second object is still the natural state of the original clothing, and the clothing still has the real physical effects of folds, pendants and the like. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

Based on the foregoing embodiments and optional embodiments, an optional implementation is provided, and is specifically described below.

In the related art, when generating a fitting image, there are generally the following ways: 1) The direct input image is diffused to generate a model, but the application of the method can cause the detail of the clothes to be changed. The generated background will be affected by the original background. 2) The clothing pictures such as warping are attached to the model body, but the attaching is not natural enough due to the mode, and the attaching is required according to key points, so that the method is limited to specific types of clothing. 3) A 3D model method is used, but the bonding is not natural enough and the manufacturing cost is too high by applying the method.

As can be seen from the above examples, at least the problem of low fitting degree between the clothing and the object in the display image generated in the related art is solved.

Based on this, an image processing method is provided in an alternative embodiment of the present invention, specifically, an image processing method is provided, taking an object as an example, fig. 7 is a schematic diagram of the image processing method provided in the alternative embodiment of the present invention, and as shown in fig. 7, the alternative embodiment of the present invention is described in detail below.

S1, acquiring an original image of a first object wearing apparel, wherein the original image is shown as (1) in FIG. 7;

the first object is a female, the clothes are casual short sleeves and casual trousers, and the female wearing the casual short sleeves and casual trousers stands in the sakura forest in the original drawing.

S2, segmenting a first object and clothes from the original image, and superposing the segmented first object and clothes into a target background image to obtain a first image, wherein the first image is shown in (2) in FIG. 7;

the target background image is a beach background, women wearing casual short sleeves and casual trousers are segmented from the original image, images of the female parts wearing casual short sleeves and casual trousers are superimposed into the beach background, and a first image is obtained, wherein a female wearing casual short sleeves and casual trousers stands on the beach edge.

S3, when the background of the first image is colored, replacing the background of the first image with a solid background, wherein the first image after replacing the background is shown in (3) in FIG. 7;

under the condition that the beach background is colored, changing the beach background into a solid background to obtain a first image after changing the background, namely that a female wearing the casual short sleeve and the casual trousers is on the solid background.

S4, performing clothing segmentation on the first image to obtain a clothing mask of the clothing, wherein the obtained clothing mask is shown in (4) in FIG. 7;

and (3) performing clothing segmentation on the images of females wearing the casual short sleeve and the casual long trousers in the solid background to obtain clothing masks, and obtaining the masks of the casual short sleeve and the casual long trousers. The female character in the image is a virtual character and is not a specific character in reality, and thus disclosure of privacy of the character is not involved.

S5, performing edge detection on the clothing mask to obtain a clothing outline, performing edge detection on the first object to obtain an object outline, wherein the outline diagram is shown in (5) in FIG. 7;

and carrying out edge detection on masks of the casual short sleeves and the casual trousers to obtain outlines of the casual short sleeves and the casual long sleeves, and carrying out edge detection on female parts to obtain outlines of the trunk of the female parts.

S6, acquiring object generation parameters, wherein the object generation parameters are acquired, and the object generation parameters comprise at least one of the following: acquiring object attribute parameters of a second object to be generated, such as darker skin color; object environment parameters of a second object to be generated, such as beach background, bright light, are obtained.

S7, inputting the first image into a diffusion generation model, taking the outline of the clothes as edge constraint of the clothes, taking the outline of the object as edge constraint of a second object, shielding the clothes by adopting a clothes mask, and generating the second object based on object generation parameters to obtain a second image, wherein the second image is shown in (6) in fig. 7.

Namely, an image of a female with darker skin color standing on the sunny beach edge can be obtained, in which the female clothing is unchanged, and the female can still be kept in the original state, so that the female can be more attached to the clothing.

Through the above-mentioned optional embodiments, the following beneficial effects can be achieved:

(1) Compared with the mode 1) in the related art, the mask is adopted to retain the original clothing details, and the background is prevented from being influenced by the original image in a background regeneration mode, so that the operability of the scheme is enhanced, and the first object has higher fitting degree of the clothing and the background.

(2) Since the original map may be a map in which an arbitrary object wears an arbitrary garment, the scheme provided by the alternative embodiment of the present invention is not limited by the clothing class, as compared to mode 2 in the related art.

(3) Because the object generation parameters are acquired, the generated second image is not constrained, and the generated image is more natural, compared with the mode 3 in the related art, the method provided by the alternative embodiment of the invention has better light and shadow fusion degree. And the 3D technical means are not used, so that the manufacturing cost is greatly reduced, and the time and the labor are saved.

(4) By the background changing regeneration mechanism, the success rate and harmony of background changing are improved.

(5) By setting the garment contours, the algorithm can be restricted from operating in areas outside the garment contours, and the garment in the second image can be ensured to be unchanged. By setting the object contour, the algorithm can be restricted to operate in the region within the object contour when generating the second object, and the second object can be generated that approximates the body height and body shape of the first object. When a second object similar to the first object in height and shape is worn, the effect of displaying the clothing with the real physical effects such as wrinkles, sagging and the like can be achieved. And because the object contour is used for constraint, the generated limbs, trunk and the like of the second object can be more reasonable, and the phenomenon of generating abnormal objects is avoided.

(6) The method can generate a clothing display diagram comparable to professional photography for clothing, can quickly generate a large number of clothing display diagrams, can be different models and different backgrounds in the large number of clothing display diagrams, is very convenient and fast, has very good image quality, and can greatly improve the display quality while ensuring low cost.

(7) The user can be provided with more convenient try-on experience, and the effect of trying on the user can be achieved by generating a hairstyle and skin color similar to those of the user and replacing the five sense organs of the user.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus a necessary general hardware platform, but that it may also be implemented by means of hardware. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present application.

Example 2

According to an embodiment of the present application, there is further provided an image processing apparatus i for implementing the image processing method i, and fig. 8 is a block diagram of the image processing apparatus i according to the embodiment of the present application, as shown in fig. 8, the apparatus includes:

a first acquiring module 802, configured to acquire a first image, where a first object in the first image wears a garment; the segmentation module 804 is connected to the first acquisition module 802, and is configured to perform garment segmentation on the first image to obtain a garment mask of the garment; a second obtaining module 806, coupled to the dividing module 804, for obtaining an object generating parameter; the processing module 808 is connected to the second obtaining module 806, and is configured to input the first image into the diffusion generating model, mask the clothing with a clothing mask, and generate a second object based on the object generating parameters, so as to obtain a second image.

Here, the first acquiring module 802, the dividing module 804, and the second acquiring module 806 correspond to the processing module 808 in steps S302 to S308 in embodiment 1, and the plurality of modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, or the above modules may also be part of an apparatus and may be run in the AR/VR device provided in embodiment 1.

Example 3

According to an embodiment of the present application, there is further provided a second image processing apparatus for implementing the second image processing method, and fig. 9 is a block diagram of the second image processing apparatus according to the embodiment of the present application, as shown in fig. 9, where the apparatus includes:

a first display module 902, configured to display a first image on an operation interface in response to an image input instruction acting on the operation interface, where a first object in the first image is wearing apparel; the second display module 904, coupled to the first display module 902, is configured to respond to a garment segmentation instruction applied to the operation interface, and display a garment mask of the garment on the operation interface, where the garment mask is obtained by performing garment segmentation on the first image; a third display module 906, coupled to the second display module 904, for displaying input object generation parameters on the operation interface in response to a parameter input command acting on the operation interface; a fourth display module 908, coupled to the third display module 906, is configured to respond to an image generation instruction applied to the operation interface, and display a second image on the operation interface, where the second image is obtained by inputting the first image into the diffusion generation model, shielding the clothing with a clothing mask, and generating a second object based on the object generation parameter.

Here, the first display module 902, the second display module 904, the third display module 906 and the fourth display module 908 correspond to steps S402 to S408 in embodiment 1, and the plurality of modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, or the above modules may also be part of an apparatus and may be run in the AR/VR device provided in embodiment 1.

Example 4

According to an embodiment of the present application, there is further provided an image processing apparatus three for implementing the above image processing method three, and fig. 10 is a block diagram of the image processing apparatus three according to an embodiment of the present application, as shown in fig. 10, including:

a display module 1002, configured to display a first image on a presentation screen of a virtual reality VR device or an augmented reality AR device, where a first object in the first image wears apparel; the receiving module 1004 is connected to the display module 1002, and is configured to receive an image processing instruction, and perform garment segmentation on the first image to obtain a garment mask of the garment; a selection module 1006, coupled to the receiving module 1004, for selecting an object generation parameter; the driving module 1008 is connected to the selecting module 1006, and is configured to drive the VR device or the AR device to display a second image, where the second image is obtained by inputting the first image into the diffusion generating model, shielding the clothing with a clothing mask, and generating a second object based on the object generating parameter.

Here, the display module 1002, the receiving module 1004, the selecting module 1006 and the driving module 1008 correspond to steps S502 to S508 in embodiment 1, and the plurality of modules are the same as the corresponding steps and the examples and application scenarios, but are not limited to those disclosed in embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, or the above modules may also be part of an apparatus and may be run in the AR/VR device provided in embodiment 1.

Example 5

According to an embodiment of the present application, there is also provided an image processing apparatus four for implementing the above image processing method four, and fig. 11 is a block diagram of the image processing apparatus four according to an embodiment of the present application, as shown in fig. 11, including:

the first calling module 1102 is configured to obtain a first image by calling a first interface, where a first object in the first image wears a garment; the second calling module 1104 is connected to the first calling module 1102, and is configured to output a clothing mask of the clothing by calling the second interface, where the clothing mask of the clothing is obtained by performing clothing segmentation on the first image; and a third calling module 1106, coupled to the second calling module 1104, configured to obtain an object generation parameter by calling the third interface, and generate a second image, where the second image is obtained by inputting the first image into the diffusion generation model, masking the clothing by using a clothing mask, and generating a second object based on the object generation parameter.

Here, the first calling module 1102, the second calling module 1104 and the third calling module 1106 correspond to steps S602 to S606 in embodiment 1, and the plurality of modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, or the above modules may also be part of an apparatus and may be run in the AR/VR device provided in embodiment 1.

Example 6

Embodiments of the present application may provide an image processing system, where the computer terminal may include an AR/VR device, a server, and a client, where the AR/VR device may be any one of an AR/VR device group. Optionally, the image processing apparatus includes: a processor; a memory coupled to the processor for providing instructions to the processor for processing the steps of: displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR device or the AR device to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

Example 7

Embodiments of the present application may provide an AR/VR device that may be any one of a group of AR/VR devices. Alternatively, in this embodiment, the AR/VR device may be replaced by a terminal device such as a mobile terminal.

Alternatively, in this embodiment, the AR/VR device may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the above-mentioned AR/VR device may execute the program codes of the following steps in the image processing method: displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR device or the AR device to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

Alternatively, fig. 12 is a block diagram of a computer terminal according to an embodiment of the present application. As shown in fig. 12, the computer terminal may include: one or more (only one is shown) processors 1202, memory 1204, a memory controller, and a peripheral interface, wherein the peripheral interface is coupled to the radio frequency module, the audio module, and the display.

The memory may be used to store software programs and modules, such as program instructions/modules corresponding to the image processing methods and apparatuses in the embodiments of the present application, and the processor executes the software programs and modules stored in the memory, thereby executing various functional applications and data processing, that is, implementing the image processing methods described above. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory remotely located with respect to the processor, the remote memory being connectable to the terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: acquiring a first image, wherein a first object in the first image is worn with clothing; performing clothing segmentation on the first image to obtain a clothing mask of clothing; obtaining object generation parameters; and inputting the first image into a diffusion generation model, shielding the clothes by adopting a clothes mask, and generating a second object based on the object generation parameters to obtain a second image.

Optionally, the above processor may further execute program code for: acquiring a first image, comprising: acquiring an original image of a first object wearing apparel; and segmenting the first object and the clothes from the original image, and overlapping the segmented first object and the segmented clothes into the target background image to obtain a first image.

Optionally, the above processor may further execute program code for: obtaining object generation parameters, including at least one of: acquiring object attribute parameters of a second object to be generated; and obtaining object environment parameters of the second object to be generated.

Optionally, the above processor may further execute program code for: performing edge detection on the clothing mask to obtain a clothing outline; inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image, wherein the method comprises the following steps: inputting the first image into a diffusion generation model, taking the outline of the clothes as edge constraint of the clothes, shielding the clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image.

Optionally, the above processor may further execute program code for: obtaining an object contour of a first object based on the first image; inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image, wherein the method comprises the following steps: inputting the first image into a diffusion generation model, taking the outline of the object as the edge constraint of the second object, shielding the clothing by adopting a clothing mask, and generating the second object based on the object generation parameters to obtain a second image.

Optionally, the above processor may further execute program code for: based on the first image, obtaining an object contour of the first object, including: converting the first image into a solid background image in case the first image is colored; and carrying out edge detection on the first object in the solid background image to obtain the object outline of the first object.

Optionally, the above processor may further execute program code for: the first object and the second object are objects with different images and/or background environments, and the objects are fit models.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: responding to an image input instruction acted on an operation interface, and displaying a first image on the operation interface, wherein a first object in the first image is worn with clothes; responding to a clothing segmentation instruction acting on an operation interface, and displaying a clothing mask of clothing on the operation interface, wherein the clothing mask is obtained by performing clothing segmentation on a first image; responding to a parameter input instruction acted on an operation interface, and displaying input object generation parameters on the operation interface; and responding to an image generation instruction acted on the operation interface, and displaying a second image on the operation interface, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR device or the AR device to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: acquiring a first image by calling a first interface, wherein a first object in the first image is worn with clothes; outputting a clothing mask of the clothing by calling the second interface, wherein the clothing mask of the clothing is obtained by carrying out clothing segmentation on the first image; and acquiring object generation parameters by calling a third interface, and generating a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on the object generation parameters.

By adopting the embodiment of the application, an image processing scheme is provided. And acquiring a first image comprising the first object wearing the clothes, performing clothes segmentation on the first image to obtain a clothes mask of the clothes, and acquiring object generation parameters. And inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters, so that the aim of obtaining the second image is finally achieved. The clothing is shielded by the clothing mask, so that the clothing part is not changed in the process of generating the second image by adopting the diffusion generation model, namely, the clothing is not deformed, the generated second object is still the natural state of the original clothing, and the clothing still has the real physical effects of folds, pendants and the like. Therefore, the method not only has the precision of generating the image according to the diffusion model, but also effectively ensures that the clothes are not deformed, the clothes and the object are still in a state of high fitting degree, the purpose of high fitting degree of the clothes and the object is achieved, and further the technical problem that the fitting degree of the clothes and the object in the display image is low when the display image is generated in the related technology is solved.

It will be appreciated by those skilled in the art that the configuration shown in fig. 12 is only illustrative, and the computer terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a mobile internet device (Mobile Internet Devices, MID), a PAD, etc. Fig. 12 is not limited to the structure of the electronic device. For example, the computer terminal may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 12, or have a different configuration than shown in FIG. 12.

Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program for instructing a terminal device to execute in association with hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.

Example 8

Embodiments of the present application also provide a computer-readable storage medium. Alternatively, in the present embodiment, the above-described computer-readable storage medium may be used to store the program code executed by the image processing method provided in the above-described embodiment 1.

Alternatively, in this embodiment, the above-mentioned computer readable storage medium may be located in any one of the AR/VR device terminals in the AR/VR device network or in any one of the mobile terminals in the mobile terminal group.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: acquiring a first image, wherein a first object in the first image is worn with clothing; performing clothing segmentation on the first image to obtain a clothing mask of clothing; obtaining object generation parameters; and inputting the first image into a diffusion generation model, shielding the clothes by adopting a clothes mask, and generating a second object based on the object generation parameters to obtain a second image.

Optionally, acquiring the first image includes: acquiring an original image of a first object wearing apparel; and segmenting the first object and the clothes from the original image, and overlapping the segmented first object and the segmented clothes into the target background image to obtain a first image.

Optionally, the object generation parameters are acquired, including at least one of: acquiring object attribute parameters of a second object to be generated; and obtaining object environment parameters of the second object to be generated.

Optionally, performing edge detection on the clothing mask to obtain a clothing outline; inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image, wherein the method comprises the following steps: inputting the first image into a diffusion generation model, taking the outline of the clothes as edge constraint of the clothes, shielding the clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image.

Optionally, obtaining an object contour of the first object based on the first image; inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters to obtain a second image, wherein the method comprises the following steps: inputting the first image into a diffusion generation model, taking the outline of the object as the edge constraint of the second object, shielding the clothing by adopting a clothing mask, and generating the second object based on the object generation parameters to obtain a second image.

Optionally, obtaining an object contour of the first object based on the first image includes: converting the first image into a solid background image in case the first image is colored; and carrying out edge detection on the first object in the solid background image to obtain the object outline of the first object.

Optionally, the first object and the second object are objects with different images and/or background environments, and the objects are fit models.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: responding to an image input instruction acted on an operation interface, and displaying a first image on the operation interface, wherein a first object in the first image is worn with clothes; responding to a clothing segmentation instruction acting on an operation interface, and displaying a clothing mask of clothing on the operation interface, wherein the clothing mask is obtained by performing clothing segmentation on a first image; responding to a parameter input instruction acted on an operation interface, and displaying input object generation parameters on the operation interface; and responding to an image generation instruction acted on the operation interface, and displaying a second image on the operation interface, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel; receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing; selecting an object generation parameter; and driving the VR device or the AR device to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on object generation parameters.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: acquiring a first image by calling a first interface, wherein a first object in the first image is worn with clothes; outputting a clothing mask of the clothing by calling the second interface, wherein the clothing mask of the clothing is obtained by carrying out clothing segmentation on the first image; and acquiring object generation parameters by calling a third interface, and generating a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding clothes by adopting a clothes mask, and generating a second object based on the object generation parameters.

The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, such as the division of the units, is merely a logical function division, and may be implemented in another manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-only memory (ROM), a random access memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.

Claims

1. An image processing method, comprising:

acquiring a first image, wherein a first object in the first image is worn with clothes;

performing clothing segmentation on the first image to obtain a clothing mask of the clothing;

obtaining object generation parameters;

and inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters to obtain a second image.

2. The method of claim 1, wherein the acquiring the first image comprises:

acquiring an original image of the first object wearing the garment;

and dividing the first object and the clothing from the original image, and overlapping the divided first object and clothing into a target background image to obtain the first image.

3. The method of claim 1, wherein the acquiring object generation parameters comprises at least one of:

acquiring object attribute parameters of the second object to be generated;

and obtaining object environment parameters of the second object to be generated.

4. The method according to claim 1, wherein the method further comprises:

performing edge detection on the clothing mask to obtain a clothing outline;

inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters to obtain a second image, wherein the method comprises the following steps:

inputting the first image into a diffusion generation model, taking the outline of the clothes as edge constraint of the clothes, adopting the clothes mask to shade the clothes, and generating a second object based on the object generation parameters to obtain the second image.

5. The method according to claim 1, wherein the method further comprises:

obtaining an object contour of the first object based on the first image;

Inputting the first image into a diffusion generation model, taking the outline of the object as the edge constraint of the second object, shielding the clothing by adopting the clothing mask, and generating the second object based on the object generation parameters to obtain the second image.

6. The method of claim 5, wherein the deriving the object contour of the first object based on the first image comprises:

converting the first image into a solid background image in case the first image is colored;

and carrying out edge detection on the first object in the solid background image to obtain an object contour of the first object.

7. The method according to any one of claims 1 to 6, wherein the first object and the second object are objects having different visual and/or background environments, the objects being fit models.

8. An image processing method, comprising:

responding to an image input instruction acted on an operation interface, and displaying a first image on the operation interface, wherein a first object in the first image is worn with clothes;

Responding to a clothing segmentation instruction acting on the operation interface, and displaying a clothing mask of the clothing on the operation interface, wherein the clothing mask is obtained by performing clothing segmentation on the first image;

responding to a parameter input instruction acted on the operation interface, and displaying input object generation parameters on the operation interface;

and responding to an image generation instruction acted on the operation interface, and displaying a second image on the operation interface, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameter.

9. An image processing method, comprising:

displaying a first image on a presentation screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein a first object in the first image is wearing apparel;

receiving an image processing instruction, and performing clothing segmentation on the first image to obtain a clothing mask of the clothing;

selecting an object generation parameter;

and driving the VR equipment or the AR equipment to display a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters.

10. An image processing method, comprising:

acquiring a first image by calling a first interface, wherein a first object in the first image is worn with clothes;

outputting a clothing mask of the clothing by calling a second interface, wherein the clothing mask of the clothing is obtained by carrying out clothing segmentation on the first image;

and acquiring object generation parameters by calling a third interface, and generating a second image, wherein the second image is obtained by inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters.

11. An image processing apparatus, comprising:

the first acquisition module is used for acquiring a first image, wherein a first object in the first image is worn with clothes;

the segmentation module is used for performing clothing segmentation on the first image to obtain a clothing mask of the clothing;

the second acquisition module is used for acquiring object generation parameters;

and the processing module is used for inputting the first image into a diffusion generation model, shielding the clothing by adopting the clothing mask, and generating a second object based on the object generation parameters to obtain a second image.

12. A computer device, comprising:

a memory storing an executable program;

a processor for executing the program, wherein the program executes the image processing method according to any one of claims 1 to 10 when executed.

13. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored executable program, wherein the executable program, when run, controls a device in which the computer-readable storage medium is located to perform the image processing method according to any one of claims 1 to 10.