CN108010038B

CN108010038B - Live-broadcast dress decorating method and device based on self-adaptive threshold segmentation

Info

Publication number: CN108010038B
Application number: CN201711376447.9A
Authority: CN
Inventors: 赵鑫; 邱学侃; 颜水成
Original assignee: Beijing Qihoo Technology Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd
Priority date: 2017-12-19
Filing date: 2017-12-19
Publication date: 2021-07-23
Anticipated expiration: 2037-12-19
Also published as: CN108010038A

Abstract

The invention discloses a live broadcast clothing dressing method, a live broadcast clothing dressing device, a live broadcast clothing dressing computing device and a computer storage medium based on self-adaptive threshold segmentation, wherein the live broadcast clothing dressing method comprises the following steps: acquiring a current frame image containing a specific object in a video in real time; performing scene segmentation processing on a current frame image to obtain foreground probability information aiming at a specific object, determining a foreground region proportion according to the foreground probability information, and performing mapping processing on the foreground probability information according to the foreground region proportion to obtain an image segmentation result corresponding to the current frame image; determining a processed foreground image according to the image segmentation result, and determining a limb area in the processed foreground image according to the processed foreground image; adding a dress decorating effect to the limb area according to dress decorating selected by a user to obtain a frame processing image; covering the frame processing image on the current frame image to obtain processed video data; and displaying the processed video data. This technical scheme can add dress effect to the people accurately.

Description

Live-broadcast dress decorating method and device based on self-adaptive threshold segmentation

Technical Field

The invention relates to the technical field of image processing, in particular to a live dress decorating method and device based on self-adaptive threshold segmentation, computing equipment and a computer storage medium.

Background

In the prior art, when a user needs to perform personalized processing on a video, for example, adding a dress effect to a character in the video, an image segmentation method is often used to perform scene segmentation processing on a frame image in the video, wherein a pixel-level segmentation effect can be achieved by using the image segmentation method based on deep learning. However, when the existing image segmentation method is used for scene segmentation processing, the proportion of the foreground image in the frame image is not considered, so when the proportion of the foreground image in the frame image is small, the existing image segmentation method is used for easily dividing the pixel points which actually belong to the edge of the foreground image into the background image, and the obtained image segmentation result has low segmentation precision and poor segmentation effect. Therefore, the image segmentation method in the prior art has the problem that the segmentation precision of image scene segmentation is low, and the obtained image segmentation result cannot be used for well and accurately adding the dress decorating effect to the body area of a person in a video, so that the obtained processed video data has poor display effect.

Disclosure of Invention

In view of the above, the present invention has been developed to provide a method, apparatus, computing device and computer storage medium for live dress based on adaptive threshold segmentation that overcome or at least partially address the above-mentioned problems.

According to an aspect of the present invention, there is provided a live dress preparation method based on adaptive threshold segmentation, the method comprising:

acquiring a current frame image containing a specific object in a video shot and/or recorded by image acquisition equipment in real time;

performing scene segmentation processing on a current frame image to obtain foreground probability information aiming at a specific object, determining a foreground region proportion according to the foreground probability information, and performing mapping processing on the foreground probability information according to the foreground region proportion to obtain an image segmentation result corresponding to the current frame image;

determining a processed foreground image according to the image segmentation result, and determining a limb area in the processed foreground image according to the processed foreground image;

adding a dress decorating effect to the limb area according to dress decorating selected by a user to obtain a frame processing image;

covering the frame processing image on the current frame image to obtain processed video data;

and displaying the processed video data.

Further, the foreground probability information records the probability that each pixel point in the current frame image belongs to the foreground image.

Further, adding a dress effect to the limb area according to dress selected by the user, and obtaining the frame processing image further comprises:

determining a basic dress effect chartlet corresponding to dress according to dress selected by a user;

extracting key information of the limb area from the limb area;

processing the basic clothing decorating effect map according to the key information of the limb area to obtain a clothing decorating effect map;

and according to the key information of the limb area, carrying out fusion processing on the dress effect map and the processed foreground image to obtain a frame processing image.

Further, the key information is key point information;

according to the key information of the limb area, the step of processing the basic clothes dressing effect map further comprises the following steps:

calculating position information between at least two key points with a symmetrical relation according to the key point information;

scaling the basic clothes dressing effect map according to the distance information in the position information; and/or rotating the base clothes decorating effect map according to the rotation angle information in the position information.

Further, according to the key information of the limb area, performing fusion processing on the dress effect map and the processed foreground image to obtain a frame processing image further comprises:

determining fusion position information corresponding to the dress effect chartlet of the clothes according to the key information of the limb area;

and according to the fusion position information, carrying out fusion processing on the dress effect paste map and the processed foreground image to obtain a frame processing image.

Further, the method further comprises:

selecting a frame image containing a specific object from a video, and determining the frame image as a frame image to be identified;

analyzing the frame image to be recognized, and determining dress of the clothes to be recommended;

and recommending dress of the clothes to be recommended to the user.

Further, analyzing the frame image to be recognized, and determining dress of the clothing to be recommended further comprises:

carrying out scene segmentation processing on a frame image to be recognized to obtain an image of a region to be recognized for a specific object;

identifying the image of the area to be identified to obtain the characteristic information of the specific object;

and matching the characteristic information with live dress by using a preset matching rule, and determining dress of the clothes to be recommended.

Further, according to the foreground probability information, determining the foreground region proportion further includes:

determining pixel points belonging to the foreground image according to the foreground probability information;

and calculating the proportion of the pixel points belonging to the foreground image in all the pixel points in the current frame image, and determining the proportion as the foreground area ratio.

Further, according to the foreground probability information, determining pixel points belonging to the foreground image further includes:

and determining the pixel points with the probability higher than a preset probability threshold in the foreground probability information as the pixel points belonging to the foreground image.

Further, mapping the foreground probability information according to the foreground region ratio to obtain an image segmentation result corresponding to the current frame image further includes:

adjusting parameters of the mapping function according to the ratio of the foreground area;

mapping the foreground probability information by using the adjusted mapping function to obtain a mapping result;

and obtaining an image segmentation result corresponding to the current frame image according to the mapping result.

Further, the slope of the mapping function in the preset defined interval is greater than a preset slope threshold.

Further, displaying the processed video data further comprises: displaying the processed video data in real time;

the method further comprises the following steps: and uploading the processed video data to a cloud server.

Further, uploading the processed video data to a cloud server further comprises:

and uploading the processed video data to a cloud video platform server so that the cloud video platform server can display the video data on a cloud video platform.

and uploading the processed video data to a cloud live broadcast server so that the cloud live broadcast server can push the video data to a client of a watching user in real time.

and uploading the processed video data to a cloud public server so that the cloud public server pushes the video data to a public attention client.

According to another aspect of the present invention, there is provided a live dress preparation apparatus based on adaptive threshold segmentation, the apparatus comprising:

the acquisition module is suitable for acquiring a current frame image containing a specific object in a video shot and/or recorded by image acquisition equipment in real time;

the segmentation module is suitable for carrying out scene segmentation processing on the current frame image to obtain foreground probability information aiming at a specific object, determining the foreground region proportion according to the foreground probability information, and carrying out mapping processing on the foreground probability information according to the foreground region proportion to obtain an image segmentation result corresponding to the current frame image;

the first determining module is suitable for determining the processed foreground image according to the image segmentation result and determining a limb area in the processed foreground image according to the processed foreground image;

the processing module is suitable for adding a clothing dressing effect to the limb area according to clothing dressing selected by a user to obtain a frame processing image;

the covering module is suitable for covering the frame processing image with the current frame image to obtain processed video data;

and the display module is suitable for displaying the processed video data.

Further, the processing module is further adapted to:

extracting key information of the limb area from the limb area;

Further, the key information is key point information;

the processing module is further adapted to:

Further, the processing module is further adapted to:

Further, the apparatus further comprises:

the selection module is suitable for selecting a frame image containing a specific object from a video and determining the frame image as a frame image to be identified;

the second determining module is suitable for analyzing the frame image to be recognized and determining dress of the clothes to be recommended;

and the recommending module is suitable for recommending the dress of the clothes to be recommended to the user.

Further, the second determination module is further adapted to:

Further, the segmentation module is further adapted to:

Further, the display module is further adapted to: displaying the processed video data in real time;

the device also includes: and the uploading module is suitable for uploading the processed video data to the cloud server.

Further, the upload module is further adapted to:

According to yet another aspect of the present invention, there is provided a computing device comprising: the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the live dress decorating method based on the adaptive threshold segmentation.

According to yet another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the above-described live dress decorating method based on adaptive threshold segmentation.

According to the technical scheme provided by the invention, the foreground probability information aiming at the specific object is mapped according to the foreground area proportion, the self-adaptive mapping of the foreground probability information is realized, the image segmentation result corresponding to the frame image can be quickly and accurately obtained by utilizing the mapped foreground probability information, the segmentation precision and the processing efficiency of image scene segmentation are effectively improved, the image scene segmentation processing mode is optimized, and the clothing dressing effect can be more accurately and quickly added to the body area of the figure in the frame image based on the obtained image segmentation result, so that the video data display effect is beautified, and the video data processing efficiency is improved.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 shows a schematic flow diagram of a live dress-up method based on adaptive threshold segmentation according to an embodiment of the present invention;

fig. 2 shows a schematic flow diagram of a live dress preparation method based on adaptive threshold segmentation according to another embodiment of the present invention;

fig. 3 is a block diagram illustrating a structure of a live dress apparatus based on adaptive threshold segmentation according to an embodiment of the present invention;

FIG. 4 shows a schematic structural diagram of a computing device according to an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Fig. 1 shows a flow chart of a live dress preparation method based on adaptive threshold segmentation according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:

step S100, acquiring a current frame image containing a specific object in a video shot and/or recorded by an image acquisition device in real time.

In this embodiment, the image capturing device takes a camera used by the terminal device as an example for description. The method comprises the steps of acquiring a current frame image of a camera of the terminal equipment when shooting a video or recording the video in real time. Since the specific object is processed by the method, only the current frame image containing the specific object is acquired when the current frame image is acquired. Wherein, the specific object can be a human body and the like. The specific object can be set by those skilled in the art according to actual needs, and is not limited herein.

Step S101, performing scene segmentation processing on a current frame image to obtain foreground probability information aiming at a specific object, determining a foreground region ratio according to the foreground probability information, and performing mapping processing on the foreground probability information according to the foreground region ratio to obtain an image segmentation result corresponding to the current frame image.

When the current frame image is subjected to scene segmentation processing, a depth learning method can be utilized. Deep learning is a method based on characterization learning of data in machine learning. An observation (e.g., an image) may be represented using a number of ways, such as a vector of intensity values for each pixel, or more abstractly as a series of edges, a specially shaped region, etc. And tasks are easier to learn from the examples using some specific representation methods. Scene segmentation processing can be carried out on the current frame image by utilizing a segmentation method of deep learning, and foreground probability information of the current frame image aiming at a specific object is obtained. Specifically, a scene segmentation network obtained by a deep learning method and the like may be used to perform scene segmentation processing on the current frame image to obtain foreground probability information of the current frame image for a specific object, where the foreground probability information records a probability that each pixel in the current frame image belongs to the foreground image, and specifically, a value range of the probability that each pixel belongs to the foreground image may be [0, 1 ].

In the present invention, the foreground image may only contain a specific object, and the background image is an image other than the foreground image in the current frame image. According to the foreground probability information, which pixel points in the current frame image belong to the foreground image, which pixel points belong to the background image, and which pixel points may belong to both the foreground image and the background image. For example, if the foreground probability information corresponding to a certain pixel point is close to 0, it is indicated that the pixel point belongs to a background image; if the foreground probability information corresponding to a certain pixel point is close to 1, the pixel point is indicated to belong to a foreground image; if the foreground probability information corresponding to a certain pixel point is close to 0.5, it is indicated that the pixel point may belong to both the foreground image and the background image.

After the foreground probability information is obtained, which pixel points in the current frame image belong to the foreground image can be determined according to the foreground probability information, so that the foreground area ratio is determined. The foreground area ratio is used for reflecting the ratio of the occupied area of the foreground image in the current frame image. Performing adaptive mapping processing on the foreground probability information according to the foreground region ratio, for example, when the foreground region ratio is smaller, for example, the foreground region ratio is 0.2, which indicates that the area occupied by the foreground image in the current frame image is smaller, the foreground probability information can be subjected to mapping processing, the smaller probability in the foreground probability information is adaptively mapped to a larger probability, and the larger probability in the foreground probability information is adaptively mapped to a smoother probability; for another example, when the foreground region occupancy is large, for example, the foreground region occupancy is 0.8, which indicates that the area occupied by the foreground image in the current frame image is large, the foreground probability information may be mapped, and the probability in the foreground probability information is adaptively mapped to be a smoother probability. After the foreground probability information is mapped, the image segmentation result corresponding to the current frame image is obtained according to the mapped foreground probability information.

And S102, determining the processed foreground image according to the image segmentation result, and determining the limb area in the processed foreground image according to the processed foreground image.

And clearly determining which pixel points in the current frame image belong to the foreground image and which pixel points belong to the background image according to the image segmentation result so as to determine the processed foreground image, and then identifying the processed foreground image so as to determine the limb area in the processed foreground image. Specifically, the image recognition method in the prior art can be used for recognizing the processed foreground image, and the trained recognition network can be used for recognizing the limb area in the processed foreground image. Because the recognition network is trained, the limb area in the foreground image after being processed can be conveniently obtained by inputting the foreground image after being processed into the recognition network. In the case of a specific object as a human body, the limb area may include a head area, an upper body area, a lower body area, a foot area, and the like.

Step S103, adding a dress effect to the limb area according to dress selected by the user to obtain a frame processing image.

Specifically, dress of clothes to be recommended may be recommended to the user, and the user may select one dress from dress of clothes to be recommended. After the limb area is determined, a dress effect is added to the limb area according to dress selected by the user, and a frame processing image is obtained. The person skilled in the art can add the dress effect according to the actual need, and the invention is not limited herein. Specifically, a frame processing image may be obtained by drawing a dress effect map corresponding to the limb area, and then performing fusion processing on the dress effect map and the processed foreground image, for example, a hat effect map corresponding to the head area may be drawn for the head area in the limb area; aiming at the upper body area in the limb area, a coat effect chartlet corresponding to the upper body area can be drawn; aiming at the lower body area in the limb area, a lower clothing effect map or a skirt effect map corresponding to the lower body area can be drawn; for a foot region in the limb region, a shoe effect map corresponding to the foot region may be drawn.

And step S104, covering the frame processing image on the current frame image to obtain processed video data.

The original current frame image is directly covered by the frame processing image, and the processed video data can be directly obtained. Meanwhile, the recorded user can also directly see the frame processing image.

When the frame processing image is obtained, the frame processing image is directly covered on the original current frame image. The covering is faster, and is generally completed within 1/24 seconds. For the user, since the time of the overlay processing is relatively short, the human eye does not perceive the process of overlaying the original current frame image in the video data. Therefore, when the processed video data is subsequently displayed, the processed video data is displayed in real time while the video data is shot and/or recorded and/or played, and a user cannot feel the display effect of covering the frame image in the video data.

Step S105 displays the processed video data.

After the processed video data is obtained, the processed video data can be displayed in real time, and a user can directly see the display effect of the processed video data.

According to the live-broadcast clothing decorating method based on self-adaptive threshold segmentation provided by the embodiment, the foreground probability information aiming at a specific object is mapped according to the foreground area proportion, the self-adaptive mapping of the foreground probability information is realized, the image segmentation result corresponding to a frame image can be quickly and accurately obtained by using the mapped foreground probability information, the segmentation precision and the processing efficiency of image scene segmentation are effectively improved, the image scene segmentation processing mode is optimized, the clothing decorating effect can be more accurately and quickly added to the body area of a person in the frame image based on the obtained image segmentation result, the video data display effect is beautified, and the video data processing efficiency is improved.

Fig. 2 is a flow chart of a live dress preparation method based on adaptive threshold segmentation according to another embodiment of the present invention, and as shown in fig. 2, the method includes the following steps:

step S200, acquiring a current frame image containing a specific object in a video shot and/or recorded by the image acquisition equipment in real time.

Step S201, performing scene segmentation processing on the current frame image to obtain foreground probability information aiming at a specific object, and determining the foreground area ratio according to the foreground probability information.

The method comprises the steps of determining pixel points belonging to a foreground image according to foreground probability information, then calculating the proportion of the pixel points belonging to the foreground image in all the pixel points in a current frame image, and determining the proportion as a foreground area ratio. Specifically, the foreground probability information records a probability for reflecting that each pixel in the current frame image belongs to the foreground image, and a value range of the probability for each pixel to belong to the foreground image may be [0, 1], so that a pixel with a probability higher than a preset probability threshold in the foreground probability information may be determined as a pixel belonging to the foreground image. The skilled person can set the preset probability threshold according to actual needs, and the setting is not limited herein. For example, when the preset probability threshold is 0.7, the pixel point with foreground probability information higher than 0.7 may be determined as the pixel point belonging to the foreground image. After the pixels belonging to the foreground image are determined, the number of the pixels belonging to the foreground image and the number of all pixels in the current frame image can be calculated, and the ratio of the number of the pixels belonging to the foreground image to the number of all pixels is the foreground region ratio.

And step S202, adjusting parameters of the mapping function according to the foreground area ratio, and performing mapping processing on the foreground probability information by using the adjusted mapping function to obtain a mapping result.

The mapping function may be used to map the foreground probability information, and a person skilled in the art may set the mapping function according to actual needs, which is not limited herein. For example, the mapping function may be a piecewise linear transformation function or a non-linear transformation function. And for different foreground area ratios, the parameters of the corresponding mapping functions are different. After the mapping function is adjusted, the foreground probability information can be used as an independent variable of the adjusted mapping function, and the obtained function value is the mapping result.

Specifically, when the foreground region occupies a smaller area, it indicates that the area occupied by the foreground image in the current frame image is smaller, and then in step S202, the parameters of the mapping function are adjusted according to the foreground region occupation ratio, so that when the foreground probability information is mapped by using the adjusted mapping function, the smaller probability in the foreground probability information can be adaptively mapped to a larger probability, and the larger probability in the foreground probability information can be adaptively mapped to a smoother probability; when the foreground region accounts for a relatively large area, which indicates that the area of the foreground image in the current frame image is relatively large, in step S202, the parameters of the mapping function are adjusted according to the foreground region accounts, so that when the adjusted mapping function is used to map the foreground probability information, the probability in the foreground probability information can be adaptively mapped to a relatively smooth probability.

And the slope of the mapping function in the preset defined interval is greater than a preset slope threshold value. A person skilled in the art may set the preset definition interval and the preset slope threshold according to actual needs, which is not limited herein, for example, when the preset definition interval is (0, 0.5) and the preset slope threshold is 1, the slope of the mapping function in the definition interval (0, 0.5) is greater than 1, so that a smaller probability in the foreground probability information can be adaptively mapped to a larger probability, for example, 0.1 is mapped to 0.3.

Taking the mapping function as a non-linear transformation function as an example, in a specific embodiment, the specific formula may be as follows:

y＝1/(1+exp(-(k*x-a)))

the foreground region proportion is a foreground region proportion, k is a first parameter, a is a second parameter, specifically, the first parameter is a parameter which needs to be adjusted according to the foreground region proportion, and the second parameter is a preset fixed parameter. Assuming that the foreground region occupancy is represented by the parameter r, k may be set to 2/r and a may be set to 4, so that the corresponding value of k may be different for different foreground region occupancies.

Step S203, according to the mapping result, obtaining the image segmentation result corresponding to the current frame image.

After the mapping result is obtained, an image segmentation result corresponding to the current frame image can be obtained according to the mapping result. Compared with the prior art, the image segmentation result corresponding to the current frame image obtained according to the mapping result has higher segmentation precision and smoother segmentation edge.

And step S204, determining the processed foreground image according to the image segmentation result, and determining the limb area in the processed foreground image according to the processed foreground image.

Step S205, according to the dress selected by the user, determining a basic dress effect map corresponding to the dress.

In order to be able to draw out the dress effect paste map conveniently and quickly, a lot of basic dress effect paste maps can be drawn in advance, so when drawing the dress effect paste map corresponding to the limb area, according to the dress selected by the user, the basic dress effect paste map corresponding to the dress is determined, then the basic dress effect paste map is processed, thereby quickly obtaining the dress effect paste map. The basic clothes decorating effect maps can comprise hat effect maps, scarf effect maps, upper garment effect maps, lower garment effect maps, skirt effect maps, shoe effect maps and the like in different clothes styles. In addition, in order to facilitate management of the basic clothing decorating effect maps, an effect map library can be established, and the basic clothing decorating effect maps are stored in the effect map library.

In step S206, key information of the limb area is extracted from the limb area.

The key information may be key point information, key area information, and/or key line information. The embodiment of the present invention is described by taking the key information as the key point information as an example, but the key information of the present invention is not limited to the key point information. The key point information is used, the processing speed and efficiency of drawing the clothes decorating effect map according to the key point information can be improved, the clothes decorating effect map can be directly drawn according to the key point information, and complex operations such as follow-up calculation and analysis on the key information are not needed. Meanwhile, the key point information is convenient to extract and accurate in extraction, so that the effect of drawing the dress effect chartlet is more accurate. Specifically, the key point information of the limb area edge can be extracted from the limb area.

Step S207, processing the basic dress effect map according to the key information of the limb area to obtain the dress effect map.

Specifically, taking the key information as the key point information as an example, after the key point information of the limb area is extracted from the limb area, the position information between at least two key points having a symmetric relationship may be calculated according to the key point information, where the position information may include distance information and rotation angle information, and then the basic clothing dressing effect map is scaled according to the distance information in the position information, and/or the basic clothing dressing effect map is rotated according to the rotation angle information in the position information. By the method, the dress effect map of the clothes can be accurately drawn.

The distance between the specific object and the image acquisition equipment is different when the specific object shoots or records a video, so that the size of the specific object in the current frame image is inconsistent, and the size of the processed foreground image determined according to the image segmentation result is also inconsistent. Taking a specific object as an example of a human body, when the distance between the human body and the image acquisition equipment is long when the video is recorded, the human body is less in the current frame image, and the processed foreground image containing the human body is also less; when the distance between the human body and the image acquisition equipment is short when the video is recorded, the human body is larger in the current frame image, and the processed foreground image containing the human body is also larger. And carrying out scaling processing on the basic clothes decorating effect chartlet according to distance information in the position information between at least two key points with a symmetrical relation so as to enable the obtained clothes decorating effect chartlet to be more suitable for the size of a specific object in the processed foreground image. For example, when the processed foreground image is small, the base clothing decorating effect map can be reduced to fit the processed foreground image; when the processed foreground image is larger, the basic clothes dressing effect map can be amplified so as to be more suitable for the processed foreground image.

In addition, considering that a specific object may not face in the forward direction in the current frame image acquired when the image capturing device captures or records a video, if a human body appears in the current frame image in a head-twisting manner, in order to make the dress-up effect map more appropriate to the processed foreground image, the base dress-up effect map also needs to be rotated. Taking the upper body area in the limb area as an example, if the connecting line of the two shoulders is calculated to rotate 15 degrees to the left, the basic coat effect map is correspondingly rotated 15 degrees to the left so as to be more suitable for the processed foreground image.

And S208, according to the key information of the limb area, carrying out fusion processing on the dress effect paste map and the processed foreground image to obtain a frame processing image.

Specifically, fusion position information corresponding to the clothing decoration effect map can be determined according to key information of the limb area, and then the clothing decoration effect map and the processed foreground image are subjected to fusion processing according to the fusion position information to obtain a frame processing image.

Step S209, the frame processing image is overlaid on the current frame image to obtain processed video data.

Step S210, displaying the processed video data.

And step S211, uploading the processed video data to a cloud server.

The processed video data can be directly uploaded to a cloud server, and specifically, the processed video data can be uploaded to one or more cloud video platform servers, such as a cloud video platform server for love art, Youkou, fast video and the like, so that the cloud video platform servers can display the video data on a cloud video platform. Or the processed video data can be uploaded to a cloud live broadcast server, and when a user at a live broadcast watching end enters the cloud live broadcast server to watch, the video data can be pushed to a watching user client in real time by the cloud live broadcast server. Or the processed video data can be uploaded to a cloud public server, and when a user pays attention to the public, the cloud public server pushes the video data to a public client; further, the cloud public number server can push video data conforming to user habits to the public number attention client according to the watching habits of users paying attention to the public numbers.

Optionally, the method may further comprise: selecting a frame image containing a specific object from the video, determining the frame image as a frame image to be identified, analyzing the frame image to be identified, determining dress of the dress to be recommended, and then recommending dress of the dress to be recommended to a user. In order to recommend live dress more accurately, frame images containing a specific object may be selected from a video at preset time intervals, considering that the specific object may replace worn clothes during shooting of the video. The skilled person can set the preset time interval according to actual needs, and the preset time interval is not limited herein. For example, the preset time interval may be set to 5 minutes.

Specifically, the frame image to be recognized may be subjected to scene segmentation processing to obtain an image of an area to be recognized for the specific object, the image of the area to be recognized is recognized to obtain feature information of the specific object, the feature information is matched with live dress by using a preset matching rule, and dress of the dress to be recommended is determined.

Taking the specific object as a human body as an example, the image of the region to be recognized of the specific object may include images of regions such as a limb region and a face region of the human body. And after the image of the area to be recognized is obtained, recognizing the image of the area to be recognized to obtain the characteristic information of the specific object. Wherein the characteristic information comprises one or more of the following information: clothing feature information, skin color feature information, age feature information, gender feature information, and body shape feature information. In order to facilitate the user to dress up in the live broadcast process, various types of live broadcast dress have been set for the user in advance, each live broadcast dress has corresponding dress up characteristic information, the characteristic information is matched with the dress up characteristic information of the live broadcast dress up by utilizing a preset matching rule, and dress up to be recommended is determined from various live broadcast dress ups. Wherein, a person skilled in the art can set the preset matching rule according to actual needs, and the preset matching rule is not limited herein. The live broadcast dress-up recommendation method and the live broadcast dress-up recommendation device combine the characteristic information of the user to recommend live broadcast dress-up, so that the recommended live broadcast dress-up can accord with the preference of the user, and the live broadcast dress-up recommendation effect is effectively improved.

According to the live-broadcast clothing decorating method based on self-adaptive threshold segmentation provided by the embodiment, parameters of mapping functions can be adjusted according to the foreground region occupation ratio, so that when the foreground region occupation ratio is different, the corresponding parameters of the mapping functions are different, and self-adaptive mapping of foreground probability information according to the foreground region occupation ratio is realized; the image segmentation result corresponding to the frame image can be quickly and accurately obtained by utilizing the mapping result, so that the segmentation precision and the processing efficiency of image scene segmentation are effectively improved, and the segmentation edge is smoother; moreover, based on the obtained image segmentation result, a dress decorating effect can be added to the limb area of a person in the frame image more accurately and rapidly, and the video data display effect is beautified; and according to the extracted key information of the limb area, the clothes dressing effect map can be accurately zoomed and rotated, so that the clothes dressing effect map is more suitable for a specific object, and the video data display effect is further improved. In addition, live-broadcast dress-up recommendation can be performed by combining with the characteristic information of the user, so that the recommended live-broadcast dress-up can accord with the preference of the user, and the live-broadcast dress-up recommendation effect is effectively improved.

Fig. 3 is a block diagram illustrating a structure of an adaptive threshold segmentation-based live dress preparation apparatus according to an embodiment of the present invention, as shown in fig. 3, the apparatus including: an acquisition module 301, a segmentation module 302, a first determination module 303, a processing module 304, an overlay module 305, and a display module 306.

The acquisition module 301 is adapted to: and acquiring a current frame image containing a specific object in a video shot and/or recorded by the image acquisition equipment in real time.

The segmentation module 302 is adapted to: the method comprises the steps of carrying out scene segmentation processing on a current frame image to obtain foreground probability information aiming at a specific object, determining a foreground region proportion according to the foreground probability information, and carrying out mapping processing on the foreground probability information according to the foreground region proportion to obtain an image segmentation result corresponding to the current frame image.

The foreground probability information records the probability of each pixel point in the current frame image belonging to the foreground image. The segmentation module 302 is further adapted to: determining pixel points belonging to the foreground image according to the foreground probability information; and calculating the proportion of the pixel points belonging to the foreground image in all the pixel points in the current frame image, and determining the proportion as the foreground area ratio. Specifically, the segmentation module 302 determines the pixel points with the probability higher than the preset probability threshold in the foreground probability information as the pixel points belonging to the foreground image.

Optionally, the segmentation module 302 is further adapted to: adjusting parameters of the mapping function according to the ratio of the foreground area; mapping the foreground probability information by using the adjusted mapping function to obtain a mapping result; and obtaining an image segmentation result corresponding to the current frame image according to the mapping result. And the slope of the mapping function in the preset defined interval is greater than a preset slope threshold value.

The first determination module 303 is adapted to: and determining the processed foreground image according to the image segmentation result, and determining the limb area in the processed foreground image according to the processed foreground image.

The processing module 304 is adapted to: and adding a dress effect to the limb area according to dress selected by the user to obtain a frame processing image.

Optionally, the processing module 304 is further adapted to: determining a basic dress effect chartlet corresponding to dress according to dress selected by a user; extracting key information of the limb area from the limb area; processing the basic clothing decorating effect map according to the key information of the limb area to obtain a clothing decorating effect map; and according to the key information of the limb area, carrying out fusion processing on the dress effect map and the processed foreground image to obtain a frame processing image.

The key information may specifically be key point information, key area information, and/or key line information. The embodiment of the present invention is described by taking key information as key point information as an example. The processing module 304 is further adapted to: calculating position information between at least two key points with a symmetrical relation according to the key point information; scaling the basic clothes dressing effect map according to the distance information in the position information; and/or rotating the base clothes decorating effect map according to the rotation angle information in the position information.

Optionally, the processing module 304 is further adapted to: determining fusion position information corresponding to the dress effect chartlet of the clothes according to the key information of the limb area; and according to the fusion position information, carrying out fusion processing on the dress effect paste map and the processed foreground image to obtain a frame processing image.

The overlay module 305 is adapted to: and covering the frame processing image on the current frame image to obtain processed video data.

The display module 306 is adapted to: and displaying the processed video data.

After the processed video data is obtained, the display module 306 can display the processed video data in real time, and a user can directly see the display effect of the processed video data.

The apparatus may further comprise: a selection module 307, a second determination module 308 and a recommendation module 309.

Wherein the selecting module 307 is adapted to: selecting a frame image containing a specific object from a video, and determining the frame image as a frame image to be identified; the second determination module 308 is adapted to: analyzing the frame image to be recognized, and determining dress of the clothes to be recommended; the recommendation module 309 is adapted to: and recommending dress of the clothes to be recommended to the user.

Optionally, the second determination module 308 is further adapted to: carrying out scene segmentation processing on a frame image to be recognized to obtain an image of a region to be recognized for a specific object; identifying the image of the area to be identified to obtain the characteristic information of the specific object; and matching the characteristic information with live dress by using a preset matching rule, and determining dress of the clothes to be recommended.

The apparatus may further comprise: the uploading module 310 is adapted to upload the processed video data to a cloud server.

The uploading module 310 may directly upload the processed video data to a cloud server, and specifically, the uploading module 310 may upload the processed video data to one or more cloud video platform servers, such as a cloud video platform server for an arcade, a super-cool, a fast video, and the like, so that the cloud video platform servers display the video data on a cloud video platform. Or the uploading module 310 may also upload the processed video data to the cloud live broadcast server, and when a user at a live broadcast watching end enters the cloud live broadcast server to watch, the cloud live broadcast server may push the video data to the watching user client in real time. Or the uploading module 310 may also upload the processed video data to a cloud public server, and when a user pays attention to the public, the cloud public server pushes the video data to a public client; further, the cloud public number server can push video data conforming to user habits to the public number attention client according to the watching habits of users paying attention to the public numbers.

According to the live dress decorating device based on self-adaptive threshold segmentation provided by the embodiment, the foreground probability information aiming at a specific object is mapped according to the foreground area proportion, the self-adaptive mapping of the foreground probability information is realized, the image segmentation result corresponding to a frame image can be quickly and accurately obtained by using the mapped foreground probability information, the segmentation precision and the processing efficiency of image scene segmentation are effectively improved, the image scene segmentation processing mode is optimized, the dress decorating effect can be more accurately and quickly added to the body area of a figure in the frame image based on the obtained image segmentation result, the video data display effect is beautified, and the video data processing efficiency is improved.

The invention also provides a nonvolatile computer storage medium, which stores at least one executable instruction, wherein the executable instruction can execute the live dress decorating method based on the adaptive threshold segmentation in any method embodiment.

Fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in fig. 4, the computing device may include: a processor (processor)402, a Communications Interface 404, a memory 406, and a Communications bus 408.

Wherein:

the processor 402, communication interface 404, and memory 406 communicate with each other via a communication bus 408.

A communication interface 404 for communicating with network elements of other devices, such as clients or other servers.

A processor 402 is configured to execute a program 410, which may specifically execute relevant steps in the above-described embodiment of the live dress preparation method based on adaptive threshold segmentation.

In particular, program 410 may include program code comprising computer operating instructions.

The processor 402 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.

And a memory 406 for storing a program 410. Memory 406 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 410 may be specifically configured to cause the processor 402 to execute the live dress preparation method based on adaptive threshold segmentation in any of the above-described method embodiments. For specific implementation of each step in the procedure 410, reference may be made to corresponding steps and corresponding descriptions in units in the above live dress preparation embodiment based on adaptive threshold segmentation, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components in accordance with embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Claims

1. A live dress-up method based on adaptive threshold segmentation, the method comprising:

performing scene segmentation processing on the current frame image to obtain foreground probability information for a specific object, determining foreground region proportion according to the foreground probability information, and performing mapping processing on the foreground probability information according to the foreground region proportion to obtain an image segmentation result corresponding to the current frame image;

adding a clothing decorating effect to the limb area according to clothing decorating selected by a user to obtain a frame processing image;

displaying the processed video data;

wherein, the mapping the foreground probability information according to the foreground region ratio to obtain the image segmentation result corresponding to the current frame image further comprises:

adjusting parameters of a mapping function according to the foreground area ratio;

2. The method of claim 1, wherein the foreground probability information records a probability for reflecting that each pixel point in the current frame image belongs to a foreground image.

3. A method according to claim 1 or 2, wherein said adding a dress effect to said limb area according to a user selected dress of a dress, resulting in a frame-processed image further comprises:

extracting key information of the limb area from the limb area;

and according to the key information of the limb area, carrying out fusion processing on the dress effect chartlet and the processed foreground image to obtain a frame processing image.

4. The method of claim 3, wherein the key information is key point information;

the processing the basic clothing decorating effect map according to the key information of the limb area further comprises:

according to the distance information in the position information, carrying out scaling treatment on the base clothes decorating effect map; and/or rotating the basic clothes dressing effect chartlet according to the rotation angle information in the position information.

5. The method according to claim 4, wherein the fusing the dress effect map and the processed foreground image according to the key information of the limb area to obtain a frame processing image further comprises:

determining fusion position information corresponding to the dress effect chartlet according to the key information of the limb area;

6. The method of claim 5, wherein the method further comprises:

selecting a frame image containing a specific object from the video, and determining the frame image as a frame image to be identified;

analyzing the frame image to be identified, and determining dress of the clothes to be recommended;

and recommending dress of the clothes to be recommended to the user.

7. The method of claim 6, wherein the analyzing the frame image to be identified and determining dress of clothing to be recommended further comprises:

carrying out scene segmentation processing on the frame image to be identified to obtain an image of a region to be identified for the specific object;

8. The method of claim 7, wherein said determining a foreground region proportion from said foreground probability information further comprises:

9. The method of claim 8, wherein said determining pixel points belonging to a foreground image according to the foreground probability information further comprises:

10. The method of claim 1, wherein a slope of the mapping function within a preset defined interval is greater than a preset slope threshold.

11. The method of claim 10, wherein the displaying the processed video data further comprises: displaying the processed video data in real time;

12. The method of claim 11, wherein the uploading the processed video data to a cloud server further comprises:

13. The method of claim 11, wherein the uploading the processed video data to a cloud server further comprises:

14. The method of claim 11, wherein the uploading the processed video data to a cloud server further comprises:

15. A live dress apparatus based on adaptive threshold segmentation, the apparatus comprising:

the segmentation module is suitable for carrying out scene segmentation processing on the current frame image to obtain foreground probability information aiming at a specific object, determining foreground region proportion according to the foreground probability information, and carrying out mapping processing on the foreground probability information according to the foreground region proportion to obtain an image segmentation result corresponding to the current frame image;

the first determining module is suitable for determining a processed foreground image according to the image segmentation result and determining a limb area in the processed foreground image according to the processed foreground image;

the processing module is suitable for adding a dress decorating effect to the limb area according to dress decorating selected by a user to obtain a frame processing image;

the covering module is suitable for covering the frame processing image on the current frame image to obtain processed video data;

the display module is suitable for displaying the processed video data;

wherein the segmentation module is further adapted to:

16. The apparatus of claim 15, wherein the foreground probability information records a probability for reflecting that each pixel in the current frame image belongs to a foreground image.

17. The apparatus of claim 15 or 16, wherein the processing module is further adapted to:

extracting key information of the limb area from the limb area;

18. The apparatus of claim 17, wherein the key information is key point information;

the processing module is further adapted to:

19. The apparatus of claim 17, wherein the processing module is further adapted to:

20. The apparatus of claim 19, wherein the apparatus further comprises:

the selection module is suitable for selecting a frame image containing a specific object from the video and determining the frame image as a frame image to be identified;

the second determining module is suitable for analyzing the frame image to be identified and determining dress of the clothes to be recommended;

and the recommending module is suitable for recommending the dress to be recommended to the user.

21. The apparatus of claim 20, wherein the second determining module is further adapted to:

22. The apparatus of claim 21, wherein the segmentation module is further adapted to:

23. The apparatus of claim 22, wherein the segmentation module is further adapted to:

24. The apparatus of claim 15, wherein a slope of the mapping function within a preset defined interval is greater than a preset slope threshold.

25. The apparatus of claim 24, wherein the display module is further adapted to: displaying the processed video data in real time;

the device further comprises: and the uploading module is suitable for uploading the processed video data to the cloud server.

26. The apparatus of claim 25, wherein the upload module is further adapted to:

27. The apparatus of claim 25, wherein the upload module is further adapted to:

28. The apparatus of claim 25, wherein the upload module is further adapted to:

29. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is configured to store at least one executable instruction that causes the processor to perform operations corresponding to the adaptive threshold segmentation based live apparel fitting method of any of claims 1-14.

30. A computer storage medium having stored therein at least one executable instruction to cause a processor to perform operations corresponding to the adaptive threshold segmentation based live apparel fitting method of any of claims 1-14.