WO2021164283A1 - Clothing color recognition method, device and system based on semantic segmentation - Google Patents

Clothing color recognition method, device and system based on semantic segmentation Download PDF

Info

Publication number
WO2021164283A1
WO2021164283A1 PCT/CN2020/121515 CN2020121515W WO2021164283A1 WO 2021164283 A1 WO2021164283 A1 WO 2021164283A1 CN 2020121515 W CN2020121515 W CN 2020121515W WO 2021164283 A1 WO2021164283 A1 WO 2021164283A1
Authority
WO
WIPO (PCT)
Prior art keywords
clothing
pictures
color recognition
portrait
clothing color
Prior art date
Application number
PCT/CN2020/121515
Other languages
French (fr)
Chinese (zh)
Inventor
王江鹏
毛晓蛟
赵文忠
章勇
曹李军
Original Assignee
苏州科达科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏州科达科技股份有限公司 filed Critical 苏州科达科技股份有限公司
Publication of WO2021164283A1 publication Critical patent/WO2021164283A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present invention relates to the technical field of image processing, in particular to a clothing color recognition method, device and system based on semantic segmentation.
  • the color recognition of clothing generally directly classifies the color of the upper and lower clothing after inputting the entire portrait into the classifier.
  • this method is not suitable for situations where the portrait is incomplete due to the shooting angle, occlusion, etc., or the pose of the portrait is complicated. The problem is difficult to judge.
  • the industry also has multi-label classification based on graph neural networks.
  • the reason for this is that the relationship between the color of the upper garment, the color of the lower garment and other attributes of the human body can be used. There is no obvious relationship between clothing style, clothing length and other attributes, leading to a certain degree of improvement in the recognition of other attributes, but it has no positive effect on the recognition of clothing color.
  • the industry also has a method based on the attention mechanism.
  • This method uses attention technology to initially locate the position of the top and bottom clothes, and then make color judgments based on these positioning features, but because the human body is not a shape that does not change Rigid body, due to its own posture changes, will cause occlusion, distortion and other phenomena.
  • video surveillance scenes cannot fully capture everyone's full body shots. The attention mechanism will be when facing these complex scenes. The problem of inaccurate positioning will result in a direct result that will affect the final classification result and cause recognition errors.
  • the existing invention patent discloses a method for segmentation of fashion clothing images based on deep learning, which can identify the semantic information of upper body clothing and lower body clothing from complex scenes, and input the source image into deep learning designed specifically for the field of fashion clothing.
  • the network is trained to automatically recognize upper body clothing, lower body clothing, and full body clothing collocations in the image.
  • the clothing key point information and key point visualization information are used to pool the global features of the image input by the image feature extraction module around the key point positions to obtain local features, which are related to the deformation of the clothing. Occlusion is irrelevant, so the recognition and segmentation accuracy of clothing is greatly improved.
  • the clothing key point information includes coordinate point information of various clothing, for example, for upper body clothing, there are coordinate points such as left collar, right collar, left sleeve, right sleeve, left hem, and lower right corner.
  • the purpose of the present invention is to provide a clothing color recognition method, device and system based on semantic segmentation, which uses a large number of labeled semantic segmentation data to perform detailed segmentation on clothing contained in portrait pictures, and extract each of the clothing contained in it.
  • the partial pictures of the clothing area are processed to form a new picture and sent to the classifier for color recognition.
  • the present invention can achieve a higher recognition success rate and recognition accuracy rate based on the same data set, especially for complex situations such as busts and occlusions in the picture, the recognition success rate is lower.
  • the recognition accuracy rate is improved even higher. For example, in the case of a bust, the present invention can achieve almost complete judgment accuracy.
  • the present invention proposes a clothing color recognition method based on semantic segmentation.
  • the clothing color recognition method includes the following steps:
  • S1 Collect a certain amount of portrait pictures with different parameters in different scenes, and annotate the collected portrait pictures.
  • the annotation content includes clothing region segmentation information annotation and human body joint point information annotation; random parameter transformation processing is performed on the labeled portrait image to generate Initial sample set;
  • S2 Create a clothing region extraction model based on the JPPNet network, the clothing region extraction model is used to combine the body joint point information annotation and the clothing region segmentation information annotation to extract the clothing region on the portrait picture; import the portrait picture in the initial sample set into the clothing The region extraction model extracts the partial pictures of each clothing region contained in the portrait pictures;
  • S3 Perform background transformation and color classification and labeling on the extracted partial images of each clothing area; perform size unification and random parameter transformation processing on the labeled partial images to generate a training sample set;
  • S4 Create a clothing color recognition model based on the classifier, and import the training sample set into the clothing color recognition model to train the clothing color recognition model;
  • S5 Collect portrait pictures containing clothing information in real time, and use a clothing region extraction model and a clothing color recognition model to recognize the clothing colors of one or more clothing regions in the portrait picture.
  • the parameters of the portrait picture include shooting parameters and human body posture parameters
  • the shooting parameters include lighting conditions, shooting scenes, shooting angles, and shooting distances;
  • the human body posture parameters include human body posture, full-body close-up, and half-body close-up.
  • the clothing region segmentation information label includes the information labels of the head, upper garment, lower garment, limbs, and foot regions;
  • the human body joint point information labeling includes the information labeling of the human body's wrist, elbow, shoulder, head, chest cavity, knee joint, and ankle joint point.
  • the random parameter transformation processing refers to random cropping, rotation, flipping, and color transformation processing on the picture.
  • step S3 performing background transformation on the extracted partial pictures of each clothing region refers to unifying the background regions in the extracted partial pictures of each clothing region into a pure white background.
  • step S3 a bilinear interpolation method is used to unify the marked partial pictures to the same size.
  • step S5 using the clothing region extraction model and the clothing color recognition model to recognize the clothing color of one or more clothing regions in the portrait picture includes the following steps:
  • S51 Collect portrait pictures containing clothing information in real time, and import the portrait pictures into the clothing region extraction model to extract partial pictures of each clothing region contained in the portrait pictures;
  • S52 Perform size unification and background transformation processing on the extracted partial pictures of each clothing area, and import the processed partial pictures of each clothing area into the clothing color recognition model to identify the corresponding clothing color.
  • the present invention proposes a clothing color recognition device based on semantic segmentation.
  • the clothing color recognition device includes:
  • the clothing region extraction model created based on the JPPNet network is used to extract the clothing region on the portrait image combined with the annotation of the human joint point information and the annotation of the clothing region segmentation information;
  • the clothing color recognition model created based on the classifier is used to recognize the clothing color in the imported partial images of each clothing area;
  • Portrait picture collection module used to collect portrait pictures with different parameters in different scenarios
  • the sample set generation module is used to perform random parameter transformation processing on the imported pictures to generate corresponding training picture sample sets;
  • the image preprocessing module is used to perform background transformation and uniform size processing on the imported pictures.
  • the present invention proposes a clothing color recognition system based on semantic segmentation.
  • the clothing color recognition system includes a memory, a processor, and a computer stored in the memory and running on the processor. program;
  • the clothing contained in the portrait image is segmented more accurately and meticulously, so as to extract the partial image of each clothing area it contains, due to the introduction of human body joint point information Label, the extracted partial pictures of each clothing area are no longer limited by factors such as incompleteness or shooting direction, and then process the partial pictures of each clothing area to form a new picture and send it to the classifier for color recognition and zoom out
  • the color recognition range is increased, and the color recognition efficiency is improved; compared with other clothing color recognition methods, the present invention can achieve a higher recognition success rate and recognition accuracy rate based on the same data set.
  • the original background area of the generated partial picture is transformed into a uniform background (for example, a pure white background), so as to avoid the problem of background interference.
  • Fig. 1 is a flowchart of a clothing color recognition method based on semantic segmentation of the present invention.
  • FIG. 2 is a diagram of the clothing color recognition steps of the present invention.
  • Fig. 3 is a schematic diagram of a specific identification scene of the present invention.
  • the present invention proposes a clothing color recognition method based on semantic segmentation.
  • the clothing color recognition method includes the following steps:
  • S1 Collect a certain amount of portrait pictures with different parameters in different scenes, and annotate the collected portrait pictures.
  • the annotation content includes clothing region segmentation information annotation and human body joint point information annotation; random parameter transformation processing is performed on the labeled portrait image to generate Initial sample set.
  • the clothing region extraction model is used to combine human body joint point information annotation and clothing region segmentation information annotation to extract the clothing region on the portrait image;
  • the portrait pictures in the sample set are imported into the clothing region extraction model, and the partial pictures of each clothing region contained in the portrait pictures are extracted from the portrait pictures.
  • the JPPNet network is a deep learning method based on Tensorflow-based human body analysis and posture estimation that is often used in the prior art.
  • S3 Perform background transformation and color classification and labeling on the extracted partial pictures of each clothing region; perform size unification and random parameter transformation processing on the labeled partial pictures to generate a training sample set.
  • S4 Create a clothing color recognition model based on the classifier, and import the training sample set into the clothing color recognition model to train the clothing color recognition model.
  • S5 Collect portrait pictures containing clothing information in real time, and use a clothing region extraction model and a clothing color recognition model to recognize the clothing colors of one or more clothing regions in the portrait picture.
  • Step 1 Generate the initial sample set
  • the labeling includes two aspects: the first aspect is the segmentation and labeling of the head, upper garment, lower garment, limbs and other areas; the second aspect is the human body’s wrists, elbows, shoulders, head, chest, knee joints , Ankle and other 15 joint points.
  • the aforementioned annotation data is the data basis for extracting the partial pictures of each clothing region contained in the portrait pictures by using the JPPNet network in the second step.
  • different garments such as human body coats, tops, and jumpsuits can be distinguished by labeling. Since the number of collected portrait images is limited, the larger the training sample data imported into the clothing color recognition model, the more types, the higher the robustness and recognition rate of the clothing color recognition model generated by the final training.
  • the present invention proposes to perform random parameter transformation processing (such as random cropping, rotation, flipping, color transformation, etc. processing on the picture) on the labeled portrait picture to generate an initial sample set.
  • Step 2 Extract a partial picture of each clothing area contained in the portrait picture
  • the type of clothing area that is finally extracted is determined by the user according to actual needs, for example, only partial pictures containing upper and lower clothes are extracted.
  • the clothing region extraction model adopts JPPNet, which assists the segmentation of different regions of the human body by using human body joint points. Thanks to this assistance, compared with common semantic segmentation models, the clothing region extraction model can greatly reduce the mis-segmentation, which can significantly enhance the generalization ability of the entire model.
  • Step 3 Generate a training sample set
  • the present invention proposes to replace the original background area with a pure white background to avoid the problem of background interference.
  • the present invention proposes to perform random parameter transformation processing on the marked partial pictures (such as random cropping, rotation, flipping, color transformation, etc.) for the pictures. Increase the number of training samples as much as possible to generate a training sample set.
  • Step 4 Create and train a clothing color recognition model
  • the training sample set generated in step 3 is imported into the classifier for color recognition to complete the training of the clothing color recognition model.
  • the training sample set can be divided into a training set and a test set according to a set ratio.
  • the training set is used to train the clothing color recognition model, and then the test set is used to verify the clothing color recognition model (such as judgment and recognition). Whether the success rate and recognition accuracy rate meet the preset requirements), the training is completed if the verification is passed, otherwise, the model parameters are adjusted to retrain the model until the verification is passed.
  • step S5 the use of the clothing region extraction model and the clothing color recognition model to recognize the clothing color of one or more clothing regions in the portrait picture includes the following step:
  • S51 Collect portrait pictures containing clothing information in real time, and import the portrait pictures into the clothing region extraction model to extract partial pictures of each clothing region included in the portrait pictures.
  • S52 Perform size unification and background transformation processing on the extracted partial pictures of each clothing area, and import the processed partial pictures of each clothing area into the clothing color recognition model to identify the corresponding clothing color.
  • the present invention analyzes the corresponding relationship between the human body joint points and the clothing area through a large number of samples, and uses the human body joint points to accurately extract each clothing area in the actual clothing recognition process. For example, in a bust photo that lacks a complete bottoms picture, the incomplete bottoms picture can be extracted from the entire portrait picture by combining the limbs, knee joints, hips, etc., and then the extracted incomplete bottoms picture Background processing and color recognition.
  • the clothing color recognition method mentioned in the present invention can effectively deal with the aforementioned complex situations, and accurately extract different clothing areas, such as the upper clothing area and the lower clothing area, based on the human body joint point information in the picture, and then extract from the upper clothing area
  • the top is taken out, the bottom is extracted from the bottom area, the background color of the top and bottom is changed, and the top or bottom part is more prominent.
  • the present invention proposes an auxiliary segmentation method using human joint points to avoid this situation to a certain extent.
  • the present invention proposes a clothing color recognition device based on semantic segmentation.
  • the clothing color recognition device includes:
  • the clothing region extraction model created based on the JPPNet network is used to combine human body joint point information annotation and clothing region segmentation information annotation to extract clothing regions on portrait pictures.
  • the clothing color recognition model created based on the classifier is used to recognize the clothing color in the imported partial pictures of each clothing area.
  • the sample set generation module is used to perform random parameter transformation processing on the imported pictures to generate a corresponding training picture sample set.
  • the image preprocessing module is used to perform background transformation and uniform size processing on the imported pictures.
  • the present invention proposes a clothing color recognition system based on semantic segmentation.
  • the clothing color recognition system includes a memory, a processor, and a computer stored in the memory and running on the processor. program.
  • the processor executes the computer program, the steps of the aforementioned clothing color recognition method are realized.
  • the foregoing program can be stored in a computer readable storage medium.
  • the execution includes The steps of the foregoing method embodiment; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

A clothing color recognition method based on semantic segmentation. The method comprises: collecting a certain number of portrait pictures having different parameters in different scenes, and marking the collected portrait pictures; performing random parameter transformation processing on the marked portrait pictures to generate an initial sample set (S1); creating a clothing region extraction model on the basis of a JPPNet network, and respectively extracting local pictures of clothing regions contained in the portrait pictures (S2); performing background transformation and color classification marking on the extracted local pictures; performing size unification and random parameter transformation processing on the marked local pictures to generate a training sample set (S3); and creating a clothing color recognition model on the basis of a classifier, and importing the training sample set into the clothing color recognition model to train the clothing color recognition model (S4). According to the solution, a higher recognition success rate and recognition accuracy can be achieved, and particularly, the improvement amplitudes of the recognition success rate and the recognition accuracy are higher under the complex conditions that only a half-length portrait exists and shielding exists in the picture.

Description

基于语义分割的服装颜色识别方法、装置和***Semantic segmentation-based clothing color recognition method, device and system
本申请要求申请号为：CN202010098415.2、申请日为2020.02.18的中国国家知识产权局的在先专利申请为优先权，该在先专利申请文本中的内容通过引用而完全加入本专利申请中。This application requires the prior patent application of the State Intellectual Property Office of China with the application number of CN202010098415. 2 and the filing date of 2020.02.18 as priority, and the content of the prior patent application text is fully incorporated into this patent application by reference .
技术领域Technical field
本发明涉及图像处理技术领域，具体而言涉及一种基于语义分割的服装颜色识别方法、装置和***。The present invention relates to the technical field of image processing, in particular to a clothing color recognition method, device and system based on semantic segmentation.
背景技术Background technique
对服装颜色的识别一般通过将整个人像输入分类器之后直接对上下衣颜色进行分类，但此种方法对由于拍摄的角度、遮挡等问题造成的人像不全的情况、或者人像的姿态复杂的情况会的难以判断的问题。The color recognition of clothing generally directly classifies the color of the upper and lower clothing after inputting the entire portrait into the classifier. However, this method is not suitable for situations where the portrait is incomplete due to the shooting angle, occlusion, etc., or the pose of the portrait is complicated. The problem is difficult to judge.
除了直接用分类器判断之外，业界也有基于图神经网络做多标签分类，这样做的原因是可以利用上衣颜色、下衣颜色和人体其它属性之间的关系，但由于衣服颜色与人体其它诸如衣服款式、衣服长短等属性并不存在明显的联系，导致在其它属性的识别上可以有某种程度的提升，但对于衣服颜色的识别效果并没有什么正面的作用。In addition to directly using the classifier to judge, the industry also has multi-label classification based on graph neural networks. The reason for this is that the relationship between the color of the upper garment, the color of the lower garment and other attributes of the human body can be used. There is no obvious relationship between clothing style, clothing length and other attributes, leading to a certain degree of improvement in the recognition of other attributes, but it has no positive effect on the recognition of clothing color.
此外业界也有基于注意力机制的方法，此方法利用注意力技术，可以初步地定位上衣和下衣所在的位置，然后根据这些定位的特征来做颜色的判断，但由于人体并非一个形状不变的刚体，由于自身姿态发生变化，会造成遮挡、扭曲等现象发生，再加上视频监控场景并不能完全拍的到每个人的完全的全身照，注意力机制在面对这些复杂的场景的时候会出现定位不准确的问题，所导致的直接结果就是会影响最终的分类结果，产生识别错误的情况。In addition, the industry also has a method based on the attention mechanism. This method uses attention technology to initially locate the position of the top and bottom clothes, and then make color judgments based on these positioning features, but because the human body is not a shape that does not change Rigid body, due to its own posture changes, will cause occlusion, distortion and other phenomena. In addition, video surveillance scenes cannot fully capture everyone's full body shots. The attention mechanism will be when facing these complex scenes. The problem of inaccurate positioning will result in a direct result that will affect the final classification result and cause recognition errors.
由于人体着装情况比较复杂，衣服的位置及区域会跟随所获得的人的图像的方式、遮挡的方式、人体姿态的变化、拍摄角度的变化等等发生很大的变化，以往的服装颜色识别方案，或者粗略定位服装的位置，或者根据人体关键点的位置，或者由整张图来直接判定。例如，现有发明专利中公开了一种基于深度学习的时尚服装图像分割方法，能够从复杂场景中识别出上身服装、下身服装的语义信息，将源图像输入专门针对时尚服装领域设计的深度学习网络进行训练，自动识别出图像中上身服装、下身服装，以及全身服装搭配。在服装局部特征提取模块中，使用服装关键点信息和关键点可视化信息对由图像特征提取模块输入的图像全局特征在关键点位置周围进行池化，得到局部特征，该局部特征与服装的变形与遮挡无关，因此极大地提升了服装的识别分割精度。其中，所述服装关键点信息包括各种服装的坐标点信息，例如对于上身服装有左领、右领、左袖、右袖、左下摆、右下角等坐标点。Because the human body dress situation is more complicated, the position and area of the clothes will follow the way of the obtained human image, the way of occlusion, the change of the body posture, the change of the shooting angle, etc., and there will be great changes. The previous clothing color recognition scheme , Or roughly locate the position of the clothing, or directly determine the position of the key points of the human body, or directly from the entire picture. For example, the existing invention patent discloses a method for segmentation of fashion clothing images based on deep learning, which can identify the semantic information of upper body clothing and lower body clothing from complex scenes, and input the source image into deep learning designed specifically for the field of fashion clothing. The network is trained to automatically recognize upper body clothing, lower body clothing, and full body clothing collocations in the image. In the clothing local feature extraction module, the clothing key point information and key point visualization information are used to pool the global features of the image input by the image feature extraction module around the key point positions to obtain local features, which are related to the deformation of the clothing. Occlusion is irrelevant, so the recognition and segmentation accuracy of clothing is greatly improved. Wherein, the clothing key point information includes coordinate point information of various clothing, for example, for upper body clothing, there are coordinate points such as left collar, right collar, left sleeve, right sleeve, left hem, and lower right corner.
但在实际应用中，这些方案都很难适应复杂情况的识别，例如在前述专利中，必须依赖服装的坐标点信息对服装区域进行定位，当某一服装区域的部分坐标点信息由于被遮挡或拍摄角度等原因导致丢失时，需要对丢失的坐标点进行拟合，除了因拟合过程计算量较大导致识别速度慢之外，拟合得到的坐标点的精度较低也使得实际应用中的最终识别效果大打折扣。However, in practical applications, these solutions are difficult to adapt to the recognition of complex situations. For example, in the aforementioned patents, the clothing area must be located by relying on the clothing coordinate point information. When part of the coordinate point information of a certain clothing area is blocked or When the shooting angle and other reasons cause the loss, it is necessary to fit the missing coordinate points. In addition to the slow recognition speed due to the large amount of calculation in the fitting process, the low accuracy of the coordinate points obtained by the fitting also makes the actual application The final recognition effect is greatly reduced.
综上所述，上述技术在颜色判断上依旧存在诸如误判、漏判等等不少的问题，导致很多 时候颜色判定不准确，很难达到实用的目的。In summary, the above technologies still have many problems in color judgment, such as misjudgment, missed judgment, etc., resulting in inaccurate color judgment in many cases, and it is difficult to achieve practical purposes.
发明内容Summary of the invention
本发明目的在于提供一种基于语义分割的服装颜色识别方法、装置和***，采用大量标注的语义分割的数据，对人像图片中所包含的服装做细致的分割，提取出其所包含的每个服装区域的局部图片，再对每个服装区域的局部图片做处理后形成新图片送入分类器中进行颜色识别。相比其他服装颜色识别方法，本发明在相同的数据集基础上，能够达到更高的识别成功率和识别准确率，尤其是对于图片中只有半身像、遮挡等复杂的情况，识别成功率和识别准确率的提升幅度更高，例如，对于半身像的情况，本发明能够做到几乎完全判断准确。The purpose of the present invention is to provide a clothing color recognition method, device and system based on semantic segmentation, which uses a large number of labeled semantic segmentation data to perform detailed segmentation on clothing contained in portrait pictures, and extract each of the clothing contained in it. The partial pictures of the clothing area are processed to form a new picture and sent to the classifier for color recognition. Compared with other clothing color recognition methods, the present invention can achieve a higher recognition success rate and recognition accuracy rate based on the same data set, especially for complex situations such as busts and occlusions in the picture, the recognition success rate is lower. The recognition accuracy rate is improved even higher. For example, in the case of a bust, the present invention can achieve almost complete judgment accuracy.
为达成上述目的，结合图1，本发明提出一种基于语义分割的服装颜色识别方法，所述服装颜色识别方法包括以下步骤：In order to achieve the above objective, in conjunction with FIG. 1, the present invention proposes a clothing color recognition method based on semantic segmentation. The clothing color recognition method includes the following steps:
S1：采集一定量不同场景下不同参数的人像图片，对采集的人像图片进行标注，标注内容包括服装区域分割信息标注和人体关节点信息标注；对标注好的人像图片进行随机参数变换处理，生成初始样本集；S1: Collect a certain amount of portrait pictures with different parameters in different scenes, and annotate the collected portrait pictures. The annotation content includes clothing region segmentation information annotation and human body joint point information annotation; random parameter transformation processing is performed on the labeled portrait image to generate Initial sample set;
S2：基于JPPNet网络创建服装区域提取模型，所述服装区域提取模型用于结合人体关节点信息标注和服装区域分割信息标注对人像图片上的服装区域进行提取；将初始样本集中的人像图片导入服装区域提取模型，从人像图片中提取出各自所包含的每个服装区域的局部图片；S2: Create a clothing region extraction model based on the JPPNet network, the clothing region extraction model is used to combine the body joint point information annotation and the clothing region segmentation information annotation to extract the clothing region on the portrait picture; import the portrait picture in the initial sample set into the clothing The region extraction model extracts the partial pictures of each clothing region contained in the portrait pictures;
S3：对提取出的每个服装区域的局部图片进行背景变换和颜色分类标注；对标注好的局部图片进行尺寸统一和随机参数变换处理，生成训练样本集；S3: Perform background transformation and color classification and labeling on the extracted partial images of each clothing area; perform size unification and random parameter transformation processing on the labeled partial images to generate a training sample set;
S4：基于分类器创建服装颜色识别模型，将训练样本集导入服装颜色识别模型以对服装颜色识别模型进行训练；S4: Create a clothing color recognition model based on the classifier, and import the training sample set into the clothing color recognition model to train the clothing color recognition model;
S5：实时采集包含有服装信息的人像图片，采用服装区域提取模型和服装颜色识别模型对人像图片中的一个或多个服装区域的服装颜色进行识别。S5: Collect portrait pictures containing clothing information in real time, and use a clothing region extraction model and a clothing color recognition model to recognize the clothing colors of one or more clothing regions in the portrait picture.
进一步的实施例中，步骤S1中，所述人像图片的参数包括拍摄参数和人体姿态参数；In a further embodiment, in step S1, the parameters of the portrait picture include shooting parameters and human body posture parameters;
所述拍摄参数包括光照条件、拍摄场景、拍摄角度、拍摄距离；The shooting parameters include lighting conditions, shooting scenes, shooting angles, and shooting distances;
所述人体姿态参数包括人体姿态、全身特写、半身特写。The human body posture parameters include human body posture, full-body close-up, and half-body close-up.
进一步的实施例中，步骤S1中，所述服装区域分割信息标注包括头部、上衣、下衣、四肢、脚部区域的信息标注；In a further embodiment, in step S1, the clothing region segmentation information label includes the information labels of the head, upper garment, lower garment, limbs, and foot regions;
所述人体关节点信息标注包括人体的手腕、肘部、肩部、头部、胸腔、膝关节、脚踝关节点的信息标注。The human body joint point information labeling includes the information labeling of the human body's wrist, elbow, shoulder, head, chest cavity, knee joint, and ankle joint point.
进一步的实施例中，所述随机参数变换处理是指，对图片进行随机裁剪、旋转、翻转、颜色变换处理。In a further embodiment, the random parameter transformation processing refers to random cropping, rotation, flipping, and color transformation processing on the picture.
进一步的实施例中，步骤S3中，所述对提取出的每个服装区域的局部图片进行背景变换是指，将提取出的每个服装区域的局部图片中的背景区域统一成纯白色背景。In a further embodiment, in step S3, performing background transformation on the extracted partial pictures of each clothing region refers to unifying the background regions in the extracted partial pictures of each clothing region into a pure white background.
进一步的实施例中，步骤S3中，采用双线性插值的方法将标注好的局部图片统一到相同的尺寸。In a further embodiment, in step S3, a bilinear interpolation method is used to unify the marked partial pictures to the same size.
进一步的实施例中，步骤S5中，所述采用服装区域提取模型和服装颜色识别模型对人像图片中的一个或多个服装区域的服装颜色进行识别包括以下步骤：In a further embodiment, in step S5, using the clothing region extraction model and the clothing color recognition model to recognize the clothing color of one or more clothing regions in the portrait picture includes the following steps:
S51：实时采集包含有服装信息的人像图片，将人像图片导入服装区域提取模型以提取出人像图片中包含的每个服装区域的局部图片；S51: Collect portrait pictures containing clothing information in real time, and import the portrait pictures into the clothing region extraction model to extract partial pictures of each clothing region contained in the portrait pictures;
S52：对提取出的每个服装区域的局部图片进行尺寸统一和背景变换处理，将处理后的每个服装区域的局部图片导入服装颜色识别模型以识别出对应的服装颜色。S52: Perform size unification and background transformation processing on the extracted partial pictures of each clothing area, and import the processed partial pictures of each clothing area into the clothing color recognition model to identify the corresponding clothing color.
基于前述服装颜色识别方法，本发明提出一种基于语义分割的服装颜色识别装置，所述服装颜色识别装置包括：Based on the foregoing clothing color recognition method, the present invention proposes a clothing color recognition device based on semantic segmentation. The clothing color recognition device includes:
基于JPPNet网络创建的服装区域提取模型，用于结合人体关节点信息标注和服装区域分割信息标注对人像图片上的服装区域进行提取；The clothing region extraction model created based on the JPPNet network is used to extract the clothing region on the portrait image combined with the annotation of the human joint point information and the annotation of the clothing region segmentation information;
基于分类器创建的服装颜色识别模型，用于对导入的每个服装区域的局部图片中的服装颜色进行识别；The clothing color recognition model created based on the classifier is used to recognize the clothing color in the imported partial images of each clothing area;
人像图片采集模块，用于采集不同场景下不同参数的人像图片；Portrait picture collection module, used to collect portrait pictures with different parameters in different scenarios;
样本集生成模块，用于对导入对图片进行随机参数变换处理，生成对应的训练图片样本集；The sample set generation module is used to perform random parameter transformation processing on the imported pictures to generate corresponding training picture sample sets;
图像预处理模块，用于对导入的图片进行背景变换、尺寸统一处理。The image preprocessing module is used to perform background transformation and uniform size processing on the imported pictures.
基于前述服装颜色识别方法，本发明提出一种基于语义分割的服装颜色识别***，所述服装颜色识别***包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序；Based on the foregoing clothing color recognition method, the present invention proposes a clothing color recognition system based on semantic segmentation. The clothing color recognition system includes a memory, a processor, and a computer stored in the memory and running on the processor. program;
所述处理器执行所述计算机程序时实现如前述的服装颜色识别方法的步骤。When the processor executes the computer program, the steps of the aforementioned clothing color recognition method are realized.
以上本发明的技术方案，与现有相比，其显著的有益效果在于：Compared with the prior art, the above technical solutions of the present invention have significant beneficial effects as follows:
(1)采用大量标注的语义分割的数据，对人像图片中所包含的服装做更加精准细致的分割，以便于提取出其所包含的每个服装区域的局部图片，由于引入了人体关节点信息标注，提取出的每个服装区域的局部图片不再受限于不完整或拍摄方向等因素，再对每个服装区域的局部图片做处理后形成新图片送入分类器中进行颜色识别，缩小了颜色识别范围，提高了颜色识别效率；相比其他服装颜色识别方法，本发明在相同的数据集基础上，能够达到更高的识别成功率和识别准确率。(1) Using a large number of labeled semantic segmentation data, the clothing contained in the portrait image is segmented more accurately and meticulously, so as to extract the partial image of each clothing area it contains, due to the introduction of human body joint point information Label, the extracted partial pictures of each clothing area are no longer limited by factors such as incompleteness or shooting direction, and then process the partial pictures of each clothing area to form a new picture and send it to the classifier for color recognition and zoom out The color recognition range is increased, and the color recognition efficiency is improved; compared with other clothing color recognition methods, the present invention can achieve a higher recognition success rate and recognition accuracy rate based on the same data set.
(2)受图片中只有半身像、遮挡等复杂的情况干扰小，例如，对于半身像的情况，本发明能够做到几乎完全判断准确。(2) There is little interference from complicated situations such as busts and occlusions in the picture. For example, for busts, the present invention can make almost complete judgments accurately.
(3)基于JPPNet网络创建服装区域提取模型，服装区域提取速度快，整体服装颜色识别时间短。(3) Based on the JPPNet network to create a clothing region extraction model, the clothing region extraction speed is fast, and the overall clothing color recognition time is short.
(4)通过对图像进行随机参数变换处理，在少量人像图片的基础上，生成大量样本图片，样本集生成效率高。(4) By performing random parameter transformation processing on the image, a large number of sample pictures are generated on the basis of a small number of portrait pictures, and the sample set generation efficiency is high.
(5)生成的局部图片的原背景区域被变换成统一背景，(例如纯白色背景)，避免背景干扰的问题。(5) The original background area of the generated partial picture is transformed into a uniform background (for example, a pure white background), so as to avoid the problem of background interference.
应当理解，前述构思以及在下面更加详细地描述的额外构思的所有组合只要在这样的构思不相互矛盾的情况下都可以被视为本公开的发明主题的一部分。另外，所要求保护的主题 的所有组合都被视为本公开的发明主题的一部分。It should be understood that all combinations of the aforementioned concepts and the additional concepts described in more detail below can be regarded as part of the inventive subject matter of the present disclosure as long as such concepts are not mutually contradictory. In addition, all combinations of the claimed subject matter are regarded as part of the inventive subject matter of the present disclosure.
结合附图从下面的描述中可以更加全面地理解本发明教导的前述和其他方面、实施例和特征。本发明的其他附加方面例如示例性实施方式的特征和/或有益效果将在下面的描述中显见，或通过根据本发明教导的具体实施方式的实践中得知。The foregoing and other aspects, embodiments and features of the teachings of the present invention can be more fully understood from the following description with reference to the accompanying drawings. Other additional aspects of the present invention, such as the features and/or beneficial effects of the exemplary embodiments, will be apparent in the following description, or learned from the practice of the specific embodiments taught in accordance with the present invention.
附图说明Description of the drawings
附图不意在按比例绘制。在附图中，在各个图中示出的每个相同或近似相同的组成部分可以用相同的标号表示。为了清晰起见，在每个图中，并非每个组成部分均被标记。现在，将通过例子并参考附图来描述本发明的各个方面的实施例，其中：The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in each figure may be represented by the same reference numeral. For the sake of clarity, not every component is labeled in every figure. Now, embodiments of various aspects of the present invention will be described by way of examples and with reference to the accompanying drawings, in which:
图1是本发明的基于语义分割的服装颜色识别方法的流程图。Fig. 1 is a flowchart of a clothing color recognition method based on semantic segmentation of the present invention.
图2是本发明的服装颜色识别步骤图。Figure 2 is a diagram of the clothing color recognition steps of the present invention.
图3是本发明的具体识别场景示意图。Fig. 3 is a schematic diagram of a specific identification scene of the present invention.
具体实施方式Detailed ways
为了更了解本发明的技术内容，特举具体实施例并配合所附图式说明如下。In order to better understand the technical content of the present invention, specific embodiments are described in conjunction with the accompanying drawings as follows.
在本公开中参照附图来描述本发明的各方面，附图中示出了许多说明的实施例。本公开的实施例不必定义在包括本发明的所有方面。应当理解，上面介绍的多种构思和实施例，以及下面更加详细地描述的那些构思和实施方式可以以很多方式中任意一种来实施，这是因为本发明所公开的构思和实施例并不限于任何实施方式。另外，本发明公开的一些方面可以单独使用，或者与本发明公开的其他方面的任何适当组合来使用。In this disclosure, various aspects of the present invention are described with reference to the accompanying drawings, in which numerous illustrated embodiments are shown. The embodiments of the present disclosure are not necessarily defined to include all aspects of the present invention. It should be understood that the various concepts and embodiments introduced above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed in the present invention are not Limited to any implementation. In addition, some aspects disclosed in the present invention can be used alone or in any appropriate combination with other aspects disclosed in the present invention.
具体实施例一Specific embodiment one
结合图1，本发明提出一种基于语义分割的服装颜色识别方法，所述服装颜色识别方法包括以下步骤：With reference to Figure 1, the present invention proposes a clothing color recognition method based on semantic segmentation. The clothing color recognition method includes the following steps:
S1：采集一定量不同场景下不同参数的人像图片，对采集的人像图片进行标注，标注内容包括服装区域分割信息标注和人体关节点信息标注；对标注好的人像图片进行随机参数变换处理，生成初始样本集。S1: Collect a certain amount of portrait pictures with different parameters in different scenes, and annotate the collected portrait pictures. The annotation content includes clothing region segmentation information annotation and human body joint point information annotation; random parameter transformation processing is performed on the labeled portrait image to generate Initial sample set.
S2：基于JPPNet网络(Joint Body Parsing&Pose Estimation Network)创建服装区域提取模型，所述服装区域提取模型用于结合人体关节点信息标注和服装区域分割信息标注对人像图片上的服装区域进行提取；将初始样本集中的人像图片导入服装区域提取模型，从人像图片中提取出各自所包含的每个服装区域的局部图片。所述JPPNet网络是现有技术中经常采用的一种基于Tensorflow的人体分析和姿态估计的深度学习方法。S2: Create a clothing region extraction model based on the JPPNet network (Joint Body Parsing & Pose Estimation Network). The clothing region extraction model is used to combine human body joint point information annotation and clothing region segmentation information annotation to extract the clothing region on the portrait image; The portrait pictures in the sample set are imported into the clothing region extraction model, and the partial pictures of each clothing region contained in the portrait pictures are extracted from the portrait pictures. The JPPNet network is a deep learning method based on Tensorflow-based human body analysis and posture estimation that is often used in the prior art.
S3：对提取出的每个服装区域的局部图片进行背景变换和颜色分类标注；对标注好的局部图片进行尺寸统一和随机参数变换处理，生成训练样本集。S3: Perform background transformation and color classification and labeling on the extracted partial pictures of each clothing region; perform size unification and random parameter transformation processing on the labeled partial pictures to generate a training sample set.
S4：基于分类器创建服装颜色识别模型，将训练样本集导入服装颜色识别模型以对服装颜色识别模型进行训练。S4: Create a clothing color recognition model based on the classifier, and import the training sample set into the clothing color recognition model to train the clothing color recognition model.
S5：实时采集包含有服装信息的人像图片，采用服装区域提取模型和服装颜色识别模型对人像图片中的一个或多个服装区域的服装颜色进行识别。S5: Collect portrait pictures containing clothing information in real time, and use a clothing region extraction model and a clothing color recognition model to recognize the clothing colors of one or more clothing regions in the portrait picture.
下面结合具体例子对前述步骤进行详细阐述。The foregoing steps are described in detail below in conjunction with specific examples.
步骤一、生成初始样本集Step 1: Generate the initial sample set
首先采集姿态、光照、场景、角度各不相同的人像图片，包括一些全身的、半身的图片，再对图片进行两个方面的标注。标注包括两个方面：第一个方面是对头部、上衣、下衣、四肢等区域的分割标注；第二个方面是对人体的手腕、肘部、肩部、头部、胸腔、膝关节、脚踝等15个关节点的标注。First, collect portrait pictures with different poses, lighting, scenes, and angles, including some full-body and half-body pictures, and then mark the pictures in two aspects. The labeling includes two aspects: the first aspect is the segmentation and labeling of the head, upper garment, lower garment, limbs and other areas; the second aspect is the human body’s wrists, elbows, shoulders, head, chest, knee joints , Ankle and other 15 joint points.
前述标注数据是步骤二中采用JPPNet网络从人像图片中提取出各自所包含的每个服装区域的局部图片的数据基础。该步骤可以通过标注的方式将人体的外套、上衣、连衣装等等不同的服装进行区分。由于采集到的人像图像的数量有限，而导入服装颜色识别模型的训练样本的数据越大，种类越多，最终训练生成的服装颜色识别模型的鲁棒性和识别率越高。为了尽可能地增大训练样本的数量，本发明提出，对标注好的人像图片进行随机参数变换处理(如对图片采用随机裁剪、旋转、翻转、颜色变换等处理)，生成初始样本集。The aforementioned annotation data is the data basis for extracting the partial pictures of each clothing region contained in the portrait pictures by using the JPPNet network in the second step. In this step, different garments such as human body coats, tops, and jumpsuits can be distinguished by labeling. Since the number of collected portrait images is limited, the larger the training sample data imported into the clothing color recognition model, the more types, the higher the robustness and recognition rate of the clothing color recognition model generated by the final training. In order to increase the number of training samples as much as possible, the present invention proposes to perform random parameter transformation processing (such as random cropping, rotation, flipping, color transformation, etc. processing on the picture) on the labeled portrait picture to generate an initial sample set.
步骤二、从人像图片中提取出各自所包含的每个服装区域的局部图片Step 2: Extract a partial picture of each clothing area contained in the portrait picture
基于JPPNet网络创建服装区域提取模型，将初始样本集中的人像图片导入服装区域提取模型中进行训练，所述服装区域提取模型用于结合人体关节点信息标注和服装区域分割信息标注对人像图片上的服装区域进行提取，最终从人像图片中提取出各自所包含的每个服装区域的局部图片。在本发明中，最终提取出的服装区域的类型由用户根据实际需求确定，例如，只提取分别包含上衣、下衣的局部图片等。Create a clothing region extraction model based on the JPPNet network, and import the portrait pictures in the initial sample set into the clothing region extraction model for training. The clothing area is extracted, and finally a partial picture of each clothing area contained in the portrait picture is extracted from the portrait picture. In the present invention, the type of clothing area that is finally extracted is determined by the user according to actual needs, for example, only partial pictures containing upper and lower clothes are extracted.
所述服装区域提取模型采用的网络为JPPNet，通过采用人体关节点对人体不同区域的分割做辅助。得益于这项辅助，相比于常见的语义分割模型来说，服装区域提取模型可以极大地减少误分割的情况，由此可以显著地增强整个模型的泛化能力。The clothing region extraction model adopts JPPNet, which assists the segmentation of different regions of the human body by using human body joint points. Thanks to this assistance, compared with common semantic segmentation models, the clothing region extraction model can greatly reduce the mis-segmentation, which can significantly enhance the generalization ability of the entire model.
步骤三、生成训练样本集Step 3: Generate a training sample set
通过语义分割模型提取出人的上衣和下衣之后，我们选择将上衣和下衣分别提取出来形成新图片，针对这些新图片采用如双线性插值等方法来统一尺寸，继而作为训练数据送入到下一阶段的分类器中进行分类。After extracting the top and bottom clothes of the person through the semantic segmentation model, we choose to extract the top and bottom clothes separately to form new pictures. For these new pictures, we use methods such as bilinear interpolation to unify the size, and then send them as training data. Go to the next stage of the classifier for classification.
在形成局部图片的过程中，考虑到背景的颜色会影响颜色分类器的效果，本发明提出将原来为背景的区域用纯白色背景进行替换，以此来避免背景干扰的问题。In the process of forming a partial picture, considering that the color of the background will affect the effect of the color classifier, the present invention proposes to replace the original background area with a pure white background to avoid the problem of background interference.
同样的，为了提高服装颜色识别模型的鲁棒性和识别率，本发明提出，对标注好的局部图片进行随机参数变换处理(如对图片采用随机裁剪、旋转、翻转、颜色变换等处理)，尽可能地增大训练样本的数量，生成训练样本集。Similarly, in order to improve the robustness and recognition rate of the clothing color recognition model, the present invention proposes to perform random parameter transformation processing on the marked partial pictures (such as random cropping, rotation, flipping, color transformation, etc.) for the pictures. Increase the number of training samples as much as possible to generate a training sample set.
步骤四、创建并训练服装颜色识别模型Step 4: Create and train a clothing color recognition model
将步骤三生成的训练样本集导入分类器中进行颜色识别，以完成对服装颜色识别模型的训练。在训练过程中，可以将训练样本集按设定比例分别划分成训练集和测试集，采用训练集对服装颜色识别模型进行训练，之后再采用测试集对服装颜色识别模型进行验证(例如判断识别成功率和识别准确率是否满足预设要求)，验证通过则训练完成，否则，调整模型参数对模型进行重新训练，直至验证通过。The training sample set generated in step 3 is imported into the classifier for color recognition to complete the training of the clothing color recognition model. In the training process, the training sample set can be divided into a training set and a test set according to a set ratio. The training set is used to train the clothing color recognition model, and then the test set is used to verify the clothing color recognition model (such as judgment and recognition). Whether the success rate and recognition accuracy rate meet the preset requirements), the training is completed if the verification is passed, otherwise, the model parameters are adjusted to retrain the model until the verification is passed.
具体实施例二Specific embodiment two
结合图2，在前述服装颜色识别模型训练成功的基础上，步骤S5中，所述采用服装区域 提取模型和服装颜色识别模型对人像图片中的一个或多个服装区域的服装颜色进行识别包括以下步骤：With reference to Figure 2, on the basis of the successful training of the aforementioned clothing color recognition model, in step S5, the use of the clothing region extraction model and the clothing color recognition model to recognize the clothing color of one or more clothing regions in the portrait picture includes the following step:
S51：实时采集包含有服装信息的人像图片，将人像图片导入服装区域提取模型以提取出人像图片中包含的每个服装区域的局部图片。S51: Collect portrait pictures containing clothing information in real time, and import the portrait pictures into the clothing region extraction model to extract partial pictures of each clothing region included in the portrait pictures.
S52：对提取出的每个服装区域的局部图片进行尺寸统一和背景变换处理，将处理后的每个服装区域的局部图片导入服装颜色识别模型以识别出对应的服装颜色。S52: Perform size unification and background transformation processing on the extracted partial pictures of each clothing area, and import the processed partial pictures of each clothing area into the clothing color recognition model to identify the corresponding clothing color.
如图3(a)中的人像图片，当采用本发明所提及的服装颜色识别方法时，首先分割提取出两张局部图片，分别只包含上衣服装和下衣服装，再针对生成的只包含有上衣服装或下衣服装的局部图片进行背景统一、尺寸统一等处理，得到如图3(b)和图3(c)的两张图片，最后基于图3(b)和图3(c)进行服装颜色识别。As shown in Figure 3(a), when the clothing color recognition method mentioned in the present invention is adopted, two partial images are first extracted by segmentation, which only contain the top and bottom clothes respectively, and then the generated ones only contain Partial pictures with upper or lower clothes are processed for background uniformity, uniform size, etc., to obtain two pictures as shown in Figure 3 (b) and Figure 3 (c), and finally based on Figure 3 (b) and Figure 3 (c) Perform clothing color recognition.
本发明通过大量样本分析出人体关节点和服装区域存在的对应关系，在实际服装识别过程中，采用人体关节点精准提取出每个服装区域。例如，在缺少完整下装图片的半身照中，结合四肢、膝关节、髋部等即可以从整个人像图片中提取出不完整的下装图片，继而对提取出的不完整的下装图片进行背景处理和颜色识别。The present invention analyzes the corresponding relationship between the human body joint points and the clothing area through a large number of samples, and uses the human body joint points to accurately extract each clothing area in the actual clothing recognition process. For example, in a bust photo that lacks a complete bottoms picture, the incomplete bottoms picture can be extracted from the entire portrait picture by combining the limbs, knee joints, hips, etc., and then the extracted incomplete bottoms picture Background processing and color recognition.
经实践证明，基于直接分类、图卷积神经网络和基于注意力机制的上下衣颜色分类的方案，在监控场景的数据上，往往只能达到60％-70％的准确率，本发明在相同的数据集上经过测试，能够达到85％的准确率，尤其是对于图片中只有半身像、遮挡等复杂的情况，以往的方法几乎会将遮挡物的颜色判定为衣服的颜色，而本发明所提出的服装颜色识别方法对于这类情况，能够将绝大多数的情况判断正确，对于半身像的情况，甚至能够做到几乎完全判断准确。Practice has proved that the color classification scheme based on direct classification, graph convolutional neural network and attention mechanism based on the top and bottom clothes can only reach 60%-70% accuracy in monitoring scene data. The present invention is in the same After testing on the data set, it can reach an accuracy of 85%, especially for complex situations such as busts and occlusions in the picture. The previous method almost judges the color of the occluder as the color of the clothes, and the present invention The proposed clothing color recognition method can correctly judge most situations for this kind of situation, and can even make the judgment almost completely accurate for bust situations.
应用场景一Application scenario one
在人员卡口场景下，由于会出现时间、光照、角度、遮挡、姿态等等复杂多变的情况，导致所获取到的人的图片***。而本发明所提及的服装颜色识别方法能够有效地处理前述复杂的情况，结合图片中的人体关节点信息精准提取出不同的服装区域，例如上装区域和下装区域，再从上装区域中提取出上衣，从下装区域中提取出下衣，更换上衣和下衣的背景颜色，更加突出上衣或下衣部分，最后针对上衣图片和下衣图片进行颜色识别，一方面可以避免因为拍摄角度或遮挡的问题导致的半身照、头肩照等人身像不全的情况的衣服颜色判别问题，从而有效地避免了误判情况，另一方面提高了颜色识别速度。In the scene of the person bayonet, due to complex and changeable situations such as time, illumination, angle, occlusion, posture, etc., the pictures of people obtained are ever-changing. The clothing color recognition method mentioned in the present invention can effectively deal with the aforementioned complex situations, and accurately extract different clothing areas, such as the upper clothing area and the lower clothing area, based on the human body joint point information in the picture, and then extract from the upper clothing area The top is taken out, the bottom is extracted from the bottom area, the background color of the top and bottom is changed, and the top or bottom part is more prominent. Finally, the color recognition of the top and bottom pictures can be avoided because of the shooting angle or The problem of occlusion leads to the problem of color identification of clothes in situations such as half-length photos, head and shoulders photos, etc., which can effectively avoid misjudgment and improve the speed of color recognition on the other hand.
应用场景二Application scenario two
拍摄的照片中经常存在非常严重的遮挡问题，例如常见的背包遮挡、手持大件物品的遮挡、拥挤人群的遮挡，其中拥挤人群的遮挡在语义分割模型中极容易出现误分割的情况。本发明提出采用人体关节点的辅助分割方法能够在一定程度上避免这种情况，当人群的上衣、下衣处于分离的状态时，我们可以采用从空白区域切割的方案来避免误分割的情况。There are often very serious occlusion problems in captured photos, such as common backpack occlusion, occlusion of hand-held large objects, and occlusion of crowded people. Among them, the occlusion of crowded people is prone to mis-segmentation in the semantic segmentation model. The present invention proposes an auxiliary segmentation method using human joint points to avoid this situation to a certain extent. When the upper and lower clothing of the crowd are in a separated state, we can adopt the scheme of cutting from a blank area to avoid mis-segmentation.
具体实施例三Specific embodiment three
基于前述服装颜色识别方法，本发明提出一种基于语义分割的服装颜色识别装置，所述服装颜色识别装置包括：Based on the foregoing clothing color recognition method, the present invention proposes a clothing color recognition device based on semantic segmentation. The clothing color recognition device includes:
(1)基于JPPNet网络创建的服装区域提取模型，用于结合人体关节点信息标注和服装 区域分割信息标注对人像图片上的服装区域进行提取。(1) The clothing region extraction model created based on the JPPNet network is used to combine human body joint point information annotation and clothing region segmentation information annotation to extract clothing regions on portrait pictures.
(2)基于分类器创建的服装颜色识别模型，用于对导入的每个服装区域的局部图片中的服装颜色进行识别。(2) The clothing color recognition model created based on the classifier is used to recognize the clothing color in the imported partial pictures of each clothing area.
(3)人像图片采集模块，用于采集不同场景下不同参数的人像图片。(3) Portrait picture collection module, used to collect portrait pictures with different parameters in different scenarios.
(4)样本集生成模块，用于对导入对图片进行随机参数变换处理，生成对应的训练图片样本集。(4) The sample set generation module is used to perform random parameter transformation processing on the imported pictures to generate a corresponding training picture sample set.
(5)图像预处理模块，用于对导入的图片进行背景变换、尺寸统一处理。(5) The image preprocessing module is used to perform background transformation and uniform size processing on the imported pictures.
具体实施例四Specific embodiment four
基于前述服装颜色识别方法，本发明提出一种基于语义分割的服装颜色识别***，所述服装颜色识别***包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序。所述处理器执行所述计算机程序时实现如前述的服装颜色识别方法的步骤。Based on the foregoing clothing color recognition method, the present invention proposes a clothing color recognition system based on semantic segmentation. The clothing color recognition system includes a memory, a processor, and a computer stored in the memory and running on the processor. program. When the processor executes the computer program, the steps of the aforementioned clothing color recognition method are realized.
本领域普通技术人员可以理解：实施上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件完成，前述的程序可以存储与一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。A person of ordinary skill in the art can understand that all or part of the steps in the above method embodiments can be implemented by a program instructing relevant hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the execution includes The steps of the foregoing method embodiment; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.
虽然本发明已以较佳实施例揭露如上，然其并非用以限定本发明。本发明所属技术领域中具有通常知识者，在不脱离本发明的精神和范围内，当可作各种的更动与润饰。因此，本发明的保护范围当视权利要求书所界定者为准。Although the present invention has been disclosed as above in preferred embodiments, it is not intended to limit the present invention. Those with ordinary knowledge in the technical field to which the present invention belongs can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to what is defined in the claims.

Claims (9)

  1. 一种基于语义分割的服装颜色识别方法，其特征在于，所述服装颜色识别方法包括以下步骤：A clothing color recognition method based on semantic segmentation, characterized in that the clothing color recognition method includes the following steps:
    S1：采集一定量不同场景下不同参数的人像图片，对采集的人像图片进行标注，标注内容包括服装区域分割信息标注和人体关节点信息标注；对标注好的人像图片进行随机参数变换处理，生成初始样本集；S1: Collect a certain amount of portrait pictures with different parameters in different scenes, and annotate the collected portrait pictures. The annotation content includes clothing region segmentation information annotation and human body joint point information annotation; random parameter transformation processing is performed on the labeled portrait image to generate Initial sample set;
    S2：基于JPPNet网络创建服装区域提取模型，所述服装区域提取模型用于结合人体关节点信息标注和服装区域分割信息标注对人像图片上的服装区域进行提取；将初始样本集中的人像图片导入服装区域提取模型，从人像图片中提取出各自所包含的每个服装区域的局部图片；S2: Create a clothing region extraction model based on the JPPNet network, the clothing region extraction model is used to combine the body joint point information annotation and the clothing region segmentation information annotation to extract the clothing region on the portrait picture; import the portrait picture in the initial sample set into the clothing The region extraction model extracts the partial pictures of each clothing region contained in the portrait pictures;
    S3：对提取出的每个服装区域的局部图片进行背景变换和颜色分类标注；对标注好的局部图片进行尺寸统一和随机参数变换处理，生成训练样本集；S3: Perform background transformation and color classification and labeling on the extracted partial images of each clothing area; perform size unification and random parameter transformation processing on the labeled partial images to generate a training sample set;
    S4：基于分类器创建服装颜色识别模型，将训练样本集导入服装颜色识别模型以对服装颜色识别模型进行训练；S4: Create a clothing color recognition model based on the classifier, and import the training sample set into the clothing color recognition model to train the clothing color recognition model;
    S5：实时采集包含有服装信息的人像图片，采用服装区域提取模型和服装颜色识别模型对人像图片中的一个或多个服装区域的服装颜色进行识别。S5: Collect portrait pictures containing clothing information in real time, and use a clothing region extraction model and a clothing color recognition model to recognize the clothing colors of one or more clothing regions in the portrait picture.
  2. 根据权利要求1所述的基于语义分割的服装颜色识别方法，其特征在于，步骤S1中，所述人像图片的参数包括拍摄参数和人体姿态参数；The clothing color recognition method based on semantic segmentation according to claim 1, wherein in step S1, the parameters of the portrait picture include shooting parameters and human body posture parameters;
    所述拍摄参数包括光照条件、拍摄场景、拍摄角度、拍摄距离；The shooting parameters include lighting conditions, shooting scenes, shooting angles, and shooting distances;
    所述人体姿态参数包括人体姿态、全身特写、半身特写。The human body posture parameters include human body posture, full-body close-up, and half-body close-up.
  3. 根据权利要求1所述的基于语义分割的服装颜色识别方法，其特征在于，步骤S1中，所述服装区域分割信息标注包括头部、上衣、下衣、四肢、脚部区域的信息标注；The clothing color recognition method based on semantic segmentation according to claim 1, characterized in that, in step S1, the clothing region segmentation information labels include information labels of head, upper clothes, lower clothes, limbs, and foot regions;
    所述人体关节点信息标注包括人体的手腕、肘部、肩部、头部、胸腔、膝关节、脚踝关节点的信息标注。The human body joint point information labeling includes the information labeling of the human body's wrist, elbow, shoulder, head, chest cavity, knee joint, and ankle joint point.
  4. 根据权利要求1所述的基于语义分割的服装颜色识别方法，其特征在于，所述随机参数变换处理是指，对图片进行随机裁剪、旋转、翻转、颜色变换处理。The clothing color recognition method based on semantic segmentation according to claim 1, wherein the random parameter transformation processing refers to random cropping, rotation, flipping, and color transformation processing on a picture.
  5. 根据权利要求1所述的基于语义分割的服装颜色识别方法，其特征在于，步骤S3中，所述对提取出的每个服装区域的局部图片进行背景变换是指，将提取出的每个服装区域的局部图片中的背景区域统一成纯白色背景。The clothing color recognition method based on semantic segmentation according to claim 1, characterized in that, in step S3, performing background transformation on the extracted partial pictures of each clothing region refers to transforming each extracted clothing The background area in the partial image of the area is unified into a pure white background.
  6. 根据权利要求1所述的基于语义分割的服装颜色识别方法，其特征在于，步骤S3中，采用双线性插值的方法将标注好的局部图片统一到相同的尺寸。The clothing color recognition method based on semantic segmentation according to claim 1, characterized in that, in step S3, a bilinear interpolation method is used to unify the marked partial pictures to the same size.
  7. 根据权利要求1所述的基于语义分割的服装颜色识别方法，其特征在于，步骤S5中，所述采用服装区域提取模型和服装颜色识别模型对人像图片中的一个或多个服装区域的服装颜色进行识别包括以下步骤：The clothing color recognition method based on semantic segmentation according to claim 1, wherein in step S5, the clothing region extraction model and the clothing color recognition model are used to determine the clothing color of one or more clothing regions in the portrait picture. The identification includes the following steps:
    S51：实时采集包含有服装信息的人像图片，将人像图片导入服装区域提取模型以提取出人像图片中包含的每个服装区域的局部图片；S51: Collect portrait pictures containing clothing information in real time, and import the portrait pictures into the clothing region extraction model to extract partial pictures of each clothing region contained in the portrait pictures;
    S52：对提取出的每个服装区域的局部图片进行尺寸统一和背景变换处理，将处理后的每个服装区域的局部图片导入服装颜色识别模型以识别出对应的服装颜色。S52: Perform size unification and background transformation processing on the extracted partial pictures of each clothing area, and import the processed partial pictures of each clothing area into the clothing color recognition model to identify the corresponding clothing color.
  8. 一种基于语义分割的服装颜色识别装置，其特征在于，所述服装颜色识别装置包括：A clothing color recognition device based on semantic segmentation, wherein the clothing color recognition device includes:
    基于JPPNet网络创建的服装区域提取模型，用于结合人体关节点信息标注和服装区域分割信息标注对人像图片上的服装区域进行提取；The clothing region extraction model created based on the JPPNet network is used to extract the clothing region on the portrait image combined with the annotation of the human joint point information and the annotation of the clothing region segmentation information;
    基于分类器创建的服装颜色识别模型，用于对导入的每个服装区域的局部图片中的服装颜色进行识别；The clothing color recognition model created based on the classifier is used to recognize the clothing color in the imported partial images of each clothing area;
    人像图片采集模块，用于采集不同场景下不同参数的人像图片；Portrait picture collection module, used to collect portrait pictures with different parameters in different scenarios;
    样本集生成模块，用于对导入对图片进行随机参数变换处理，生成对应的训练图片样本集；The sample set generation module is used to perform random parameter transformation processing on the imported pictures to generate corresponding training picture sample sets;
    图像预处理模块，用于对导入的图片进行背景变换、尺寸统一处理。The image preprocessing module is used to perform background transformation and uniform size processing on the imported pictures.
  9. 一种基于语义分割的服装颜色识别***，其特征在于，所述服装颜色识别***包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序；A clothing color recognition system based on semantic segmentation, wherein the clothing color recognition system includes a memory, a processor, and a computer program stored in the memory and running on the processor;
    所述处理器执行所述计算机程序时实现如权利要求1-7任意一项中所述的服装颜色识别方法的步骤。When the processor executes the computer program, the steps of the clothing color recognition method according to any one of claims 1-7 are realized.
PCT/CN2020/121515 2020-02-18 2020-10-16 Clothing color recognition method, device and system based on semantic segmentation WO2021164283A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010098415.2A CN111325806A (en) 2020-02-18 2020-02-18 Clothing color recognition method, device and system based on semantic segmentation
CN202010098415.2 2020-02-18

Publications (1)

Publication Number Publication Date
WO2021164283A1 true WO2021164283A1 (en) 2021-08-26

Family

ID=71172768

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/121515 WO2021164283A1 (en) 2020-02-18 2020-10-16 Clothing color recognition method, device and system based on semantic segmentation

Country Status (2)

Country Link
CN (1) CN111325806A (en)
WO (1) WO2021164283A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113848736A (en) * 2021-09-13 2021-12-28 青岛海尔科技有限公司 Clothes information processing method and equipment based on intelligent wardrobe
CN113919998A (en) * 2021-10-14 2022-01-11 天翼数字生活科技有限公司 Image anonymization method based on semantic and attitude map guidance
CN113963374A (en) * 2021-10-19 2022-01-21 中国石油大学(华东) Pedestrian attribute identification method based on multi-level features and identity information assistance
CN114048489A (en) * 2021-09-01 2022-02-15 广东智媒云图科技股份有限公司 Human body attribute data processing method and device based on privacy protection

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111325806A (en) * 2020-02-18 2020-06-23 苏州科达科技股份有限公司 Clothing color recognition method, device and system based on semantic segmentation
CN112419249B (en) * 2020-11-12 2022-09-06 厦门市美亚柏科信息股份有限公司 Special clothing picture conversion method, terminal device and storage medium
CN112528855B (en) * 2020-12-11 2021-09-03 南方电网电力科技股份有限公司 Electric power operation dressing standard identification method and device
CN112990012A (en) * 2021-03-15 2021-06-18 深圳喜为智慧科技有限公司 Tool color identification method and system under shielding condition
CN113516062B (en) * 2021-06-24 2021-11-26 深圳开思信息技术有限公司 Customer identification method and system for automobile repair shop
CN114201681A (en) * 2021-12-13 2022-03-18 支付宝(杭州)信息技术有限公司 Method and device for recommending clothes
CN114093011B (en) * 2022-01-12 2022-05-06 北京新氧科技有限公司 Hair classification method, device, equipment and storage medium
CN117409208B (en) * 2023-12-14 2024-03-08 武汉纺织大学 Real-time clothing image semantic segmentation method and system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106227827A (en) * 2016-07-25 2016-12-14 华南师范大学 Image of clothing foreground color feature extracting method and costume retrieval method and system
CN107766861A (en) * 2017-11-14 2018-03-06 深圳码隆科技有限公司 The recognition methods of character image clothing color, device and electronic equipment
CN108229288A (en) * 2017-06-23 2018-06-29 北京市商汤科技开发有限公司 Neural metwork training and clothes method for detecting color, device, storage medium, electronic equipment
CN109325952A (en) * 2018-09-17 2019-02-12 上海宝尊电子商务有限公司 Fashion clothing image partition method based on deep learning
CN110263605A (en) * 2018-07-18 2019-09-20 桂林远望智能通信科技有限公司 Pedestrian's dress ornament color identification method and device based on two-dimension human body guise estimation
CN111325806A (en) * 2020-02-18 2020-06-23 苏州科达科技股份有限公司 Clothing color recognition method, device and system based on semantic segmentation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106227827A (en) * 2016-07-25 2016-12-14 华南师范大学 Image of clothing foreground color feature extracting method and costume retrieval method and system
CN108229288A (en) * 2017-06-23 2018-06-29 北京市商汤科技开发有限公司 Neural metwork training and clothes method for detecting color, device, storage medium, electronic equipment
CN107766861A (en) * 2017-11-14 2018-03-06 深圳码隆科技有限公司 The recognition methods of character image clothing color, device and electronic equipment
CN110263605A (en) * 2018-07-18 2019-09-20 桂林远望智能通信科技有限公司 Pedestrian's dress ornament color identification method and device based on two-dimension human body guise estimation
CN109325952A (en) * 2018-09-17 2019-02-12 上海宝尊电子商务有限公司 Fashion clothing image partition method based on deep learning
CN111325806A (en) * 2020-02-18 2020-06-23 苏州科达科技股份有限公司 Clothing color recognition method, device and system based on semantic segmentation

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LIANG, XIAODAN ET AL.: "Look into Person:Joint Body Parsing & Pose Estimation Network and a New Benchmark", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 41, no. 4, 30 April 2019 (2019-04-30), XP011712946, DOI: 10.1109/TPAMI.2018.2820063 *
XULUHONGSHANG: "(non-official translation: FashionAI", CSDN BLOG-HTTPS://BLOG.CSDN.NET/XULUHONGSHANG/ARTICLE/DETAILS/80616331, 7 June 2018 (2018-06-07) *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114048489A (en) * 2021-09-01 2022-02-15 广东智媒云图科技股份有限公司 Human body attribute data processing method and device based on privacy protection
CN113848736A (en) * 2021-09-13 2021-12-28 青岛海尔科技有限公司 Clothes information processing method and equipment based on intelligent wardrobe
CN113919998A (en) * 2021-10-14 2022-01-11 天翼数字生活科技有限公司 Image anonymization method based on semantic and attitude map guidance
CN113919998B (en) * 2021-10-14 2024-05-14 天翼数字生活科技有限公司 Picture anonymizing method based on semantic and gesture graph guidance
CN113963374A (en) * 2021-10-19 2022-01-21 中国石油大学(华东) Pedestrian attribute identification method based on multi-level features and identity information assistance

Also Published As

Publication number Publication date
CN111325806A (en) 2020-06-23

Similar Documents

Publication Publication Date Title
WO2021164283A1 (en) Clothing color recognition method, device and system based on semantic segmentation
Wang et al. Deep 3D human pose estimation: A review
US10963041B2 (en) Gesture recognition using multi-sensory data
Wetzler et al. Rule of thumb: Deep derotation for improved fingertip detection
Tang et al. Facial landmark detection by semi-supervised deep learning
Liu et al. LDCNet: limb direction cues-aware network for flexible human pose estimation in industrial behavioral biometrics systems
WO2021082692A1 (en) Pedestrian picture labeling method and device, storage medium, and intelligent apparatus
CN112101208A (en) Feature series fusion gesture recognition method and device for elderly people
CN106952312B (en) Non-identification augmented reality registration method based on line feature description
JP2018147313A (en) Object attitude estimating method, program and device
Das et al. Action recognition based on a mixture of RGB and depth based skeleton
Nguyen et al. Combined YOLOv5 and HRNet for high accuracy 2D keypoint and human pose estimation
CN109166172B (en) Clothing model construction method and device, server and storage medium
Vasconcelos et al. Methods to automatically build point distribution models for objects like hand palms and faces represented in images
CN111881841A (en) Face detection and recognition method based on binocular vision
CN102496174A (en) Method for generating face sketch index for security monitoring
Bourbakis et al. Skin-based face detection-extraction and recognition of facial expressions
Aonty et al. Multi-Person Pose Estimation Using Group-Based Convolutional Neural Network Model
Li et al. RaP-Net: A region-wise and point-wise weighting network to extract robust features for indoor localization
Ye et al. Human motion analysis based on extraction of skeleton and dynamic time warping algorithm using RGBD camera
Matsumoto et al. Automatic human pose annotation for loose-fitting clothes
Chen et al. Zeropose: Cad-model-based zero-shot pose estimation
Zhang et al. A review of human action recognition in video
Matsumoto et al. Human pose annotation using a motion capture system for loose-fitting clothes
Rong et al. RGB-D hand pose estimation using fourier descriptor

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20920283

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20920283

Country of ref document: EP

Kind code of ref document: A1