WO2021051887A1 - 一种困难样本筛选方法及装置 - Google Patents

一种困难样本筛选方法及装置 Download PDF

Info

Publication number
WO2021051887A1
WO2021051887A1 PCT/CN2020/094109 CN2020094109W WO2021051887A1 WO 2021051887 A1 WO2021051887 A1 WO 2021051887A1 CN 2020094109 W CN2020094109 W CN 2020094109W WO 2021051887 A1 WO2021051887 A1 WO 2021051887A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target area
missed
area image
label
Prior art date
Application number
PCT/CN2020/094109
Other languages
English (en)
French (fr)
Inventor
马贤忠
董维山
江浩
胡皓瑜
范一磊
Original Assignee
初速度（苏州）科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 初速度（苏州）科技有限公司 filed Critical 初速度（苏州）科技有限公司
Publication of WO2021051887A1 publication Critical patent/WO2021051887A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features

Definitions

  • the present invention relates to the technical field of intelligent driving, and in particular to a method and device for screening difficult samples.
  • Deep learning relies on a large amount of training data or samples, but when the number of samples reaches a certain scale, the potential of different newly added sample images to improve the performance of the model is different.
  • difficult samples are samples that include missed targets and falsely detected targets, which are valuable data for improving the performance of the target detection model.
  • object detection object detection
  • difficult samples are samples that include missed targets and falsely detected targets, which are valuable data for improving the performance of the target detection model.
  • the present invention provides a method and device for screening difficult samples to realize automatic screening of difficult samples.
  • the specific technical solutions are as follows:
  • an embodiment of the present invention provides a method for screening difficult samples, including:
  • the target detection model is: Containing the area where the target is located and the confidence that the detected target is located in the area where the target exists, the first missed target area image is: an image of an area with a corresponding confidence that is lower than a preset threshold;
  • the image to be screened that contains at least one first missed target area image whose corresponding target label is a missed label is determined as a difficult sample image.
  • the step of determining the target label corresponding to each first missed target area image based on the image feature of each first missed target area image and the corresponding relationship established in advance includes:
  • the target label corresponding to each first missed target area image is determined.
  • the step of determining the candidate label corresponding to each first missed target area image based on the image feature of each first missed target area image and the corresponding relationship established in advance includes:
  • a candidate label corresponding to each first missed target region image is determined.
  • the step of determining a candidate label corresponding to each first missed target area image based on the similarity value includes:
  • the first preset number of labels in the label queue corresponding to the first missed target area image are determined as candidate labels corresponding to the suspected target area image.
  • the step of determining the target label corresponding to each first missed target area image based on the candidate label corresponding to each first missed target area image includes:
  • the meeting the preset statistical condition includes: greater than a preset number threshold, or a ratio of the total number of candidate tags corresponding to the corresponding first missed target area image Greater than the preset ratio threshold;
  • the target label corresponding to the first missed target area image is a non-missed label.
  • the step of using a pre-established target detection model to detect each obtained image to be screened, and determining the image to be screened that includes at least one first missed target region image includes:
  • the area image corresponding to the candidate target area of the rectangular area will be determined as the first missed target area image
  • the area image corresponding to the smallest rectangular area containing the candidate target area is determined as the first missed target area image to determine the image that contains at least one first missed target area The image to be filtered.
  • the method before the step of determining the candidate label corresponding to each first missed target area image based on the image feature of each first missed target area image and the corresponding relationship established in advance, the method further include:
  • the annotation information includes: the annotation location information of the area where the target contained in the corresponding establishment image is located;
  • each established image is detected, and each established image including at least one second missed target area image and detection position information corresponding to the at least one second missed target area image are determined, wherein ,
  • the at least one second missed-detected target area image is an area image with a corresponding confidence level lower than the preset threshold;
  • the The second missed detection of the label corresponding to the target area image For each second missed target area image, based on the detection location information corresponding to the second missed target area image, and the label location information in the label information corresponding to the established image where the second missed target area image is located, the The second missed detection of the label corresponding to the target area image to establish the corresponding relationship.
  • the second missed target area image is based on the detection position information corresponding to the second missed target area image, and the second missed target area image is located in the annotation information corresponding to the established image
  • the step of marking the position information and determining the label corresponding to the second missed target area image to establish the corresponding relationship includes:
  • the The intersection ratio between the label frame and the detection frame corresponding to the second missed target area image For each second missed target area image, based on the detection location information corresponding to the second missed target area image, and the label location information in the label information corresponding to the established image where the second missed target area image is located, the The intersection ratio between the label frame and the detection frame corresponding to the second missed target area image;
  • intersection ratio corresponding to the second missed target area image is not less than the preset intersection ratio threshold, determine that the label corresponding to the second missed target area image is the missed label
  • intersection ratio corresponding to the second missed target area image is less than the preset intersection ratio threshold, the label corresponding to the second missed target area image is determined to be a non-missed label to establish the corresponding relationship.
  • an embodiment of the present invention provides a difficult sample screening device, including:
  • the first determining module is configured to use a pre-established target detection model to detect each obtained image to be screened, and determine the image to be screened containing at least one first missed target area image, wherein the target detection
  • the model is: used to detect the area where the target is contained in the image and determine the confidence of the existence of the target in the area where the detected target is located, and the first missed target area image is: an image of an area with a corresponding confidence lower than a preset threshold;
  • the second determining module is configured to perform image feature extraction on each first missed target area image, and determine the image feature of each first missed target area image;
  • the third determining module is configured to determine the target label corresponding to each first missed target area image based on the image feature of each first missed target area image and the pre-established corresponding relationship, wherein the corresponding relationship includes : Correspondence between the image features of the annotated images and their corresponding labels;
  • the fourth determining module is configured to determine the image to be screened containing at least one first missed target area image corresponding to the missed label as the difficult sample image.
  • the third determining module includes:
  • the first determining unit is configured to determine the candidate label corresponding to each first missed target area image based on the image feature of each first missed target area image and the corresponding relationship established in advance;
  • the second determining unit is configured to determine the target label corresponding to each first missed target area image based on the candidate label corresponding to each first missed target area image.
  • the first determining unit includes:
  • the first determining sub-module is configured to determine the first missed target based on the image feature of the first missed target area image and the image feature of each marked image for each first missed target area image The similarity value between the regional image and each marked image;
  • the second determining sub-module is configured to determine, based on the similarity value, a candidate label corresponding to each first missed target area image.
  • the second determining sub-module is specifically configured to, for each first missed target area image, according to the similarity value between the first missed target area image and each marked image, which is larger In the smallest order, arrange the labels corresponding to each marked image to obtain the label queue corresponding to the first missed target area image;
  • the first preset number of labels in the label queue corresponding to the first missed target area image are determined as candidate labels corresponding to the suspected target area image.
  • the second determining unit is specifically configured to, for each first missed target area image, count the candidate labels corresponding to the first missed target area image, which are the candidate labels of the missed label The quantity of, as the first quantity;
  • the meeting the preset statistical condition includes: greater than a preset number threshold, or a ratio of the total number of candidate tags corresponding to the corresponding first missed target area image Greater than the preset ratio threshold;
  • the target label corresponding to the first missed target area image is a non-missed label.
  • the first determining module is specifically configured to use a pre-established target detection model to detect each obtained image to be screened, determine the image to be screened containing at least one suspected target area, and determine each A confidence level corresponding to the suspected target area;
  • the area image corresponding to the candidate target area of the rectangular area will be determined as the first missed target area image
  • the area image corresponding to the smallest rectangular area containing the candidate target area is determined as the first missed target area image to determine the image that contains at least one first missed target area The image to be filtered.
  • the device further includes:
  • the relationship establishment module is configured to establish a corresponding relationship before determining the candidate label corresponding to each first missed target area image based on the image characteristics of each first missed target area image and the pre-established corresponding relationship
  • the process of, wherein the relationship establishment module includes:
  • the obtaining unit is configured to obtain the establishment image and the annotation information corresponding to each establishment image, wherein the annotation information includes: the annotation location information of the area where the target contained in the corresponding establishment image is located;
  • the third determining unit is configured to use the target detection model to detect each established image, and determine each established image including at least one second missed target area image and the at least one second missed target area Detection position information corresponding to the image, wherein the at least one second missed-detected target area image is an area image with a corresponding confidence level lower than the preset threshold;
  • the fourth determining unit is configured to perform image feature extraction on each second missed target area image, and determine the image feature of each second missed target area image;
  • the fifth determining unit is configured to, for each second missed target area image, based on the detection position information corresponding to the second missed target area image, and the annotation information corresponding to the established image where the second missed target area image is located To determine the label corresponding to the second miss-detected target area image in order to establish and obtain the corresponding relationship.
  • the fifth determining unit is specifically configured to, for each second missed target area image, based on the detection position information corresponding to the second missed target area image, and the second missed target area image
  • the label location information in the label information corresponding to the established image is determined to determine the intersection ratio between the label frame and the detection frame corresponding to the second missed-detected target area image;
  • intersection ratio corresponding to the second missed target area image is not less than the preset intersection ratio threshold, determine that the label corresponding to the second missed target area image is the missed label
  • intersection ratio corresponding to the second missed target area image is less than the preset intersection ratio threshold, the label corresponding to the second missed target area image is determined to be a non-missed label to establish the corresponding relationship.
  • the difficult sample screening method and device can use a pre-established target detection model to detect each obtained image to be screened, and determine that it contains at least one first missed target
  • the to-be-screened image of the area image where the target detection model is: used to detect the area of the target contained in the image and determine the confidence of the existence of the target in the area of the detected target, the first missed target area image is: the corresponding confidence Area images below a preset threshold; perform image feature extraction on each first missed target area image to determine the image features of each first missed target area image; images based on each first missed target area image Feature and the pre-established correspondence relationship, determine the target label corresponding to each first missed target area image, where the correspondence relationship includes the correspondence relationship between the image features of the labeled image and the corresponding label; it will contain at least one corresponding
  • the target label is the image to be screened of the first missed target area image of the missed label, and is determined to be the difficult sample image.
  • the first missed target area image of the label contains the missed target, and the image to be screened that contains at least one first missed target area image whose corresponding target label is the missed label is determined as a difficult sample image.
  • the image of the missed target area is extracted.
  • the content of the partial image block of the screened image is used as the main body of the search to avoid the interference of irrelevant information, effectively improve the accuracy of the search and recognition, and save the memory and increase the speed, so as to realize the automatic screening of difficult samples.
  • any product or method of the present invention does not necessarily need to achieve all the advantages described above at the same time.
  • each first missed target is determined
  • the target label corresponding to the area image where the target label may include a missed detection label indicating that the first missed detection target area image contains the missed target; further, the corresponding target label can be considered as the first missed detection label of the missed detection target.
  • the detected target area image contains the missed target, and the image to be screened that contains at least one first missed target area image whose corresponding target label is the missed detection label is determined as a difficult sample image.
  • the image of the missed target area is extracted.
  • the content of the partial image block of the screened image is used as the main body of the search to avoid the interference of irrelevant information, effectively improve the accuracy of the search and recognition, and save the memory and increase the speed, so as to realize the automatic screening of difficult samples.
  • the target label corresponding to each first missed target area image you can first determine the similarity value between the first missed target area image and each marked image, and then, for each first missed target area image The detection target area image, based on its corresponding similarity value, determines the labels corresponding to the previously preset number of labeled images that are most similar to the first missed detection target area image, and the labels determined above are used as the first The candidate label corresponding to the missed target area image; based on the candidate label corresponding to the first missed target area image, the target label corresponding to the first missed target area image is determined to improve the determined target label to a certain extent accuracy.
  • FIG. 1 is a schematic flowchart of a method for screening difficult samples according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a process for establishing a corresponding relationship according to an embodiment of the present invention
  • FIG. 3 is a schematic structural diagram of a difficult sample screening device provided by an embodiment of the present invention.
  • the present invention provides a method and device for screening difficult samples to realize automatic screening of difficult samples.
  • the embodiments of the present invention will be described in detail below.
  • FIG. 1 is a schematic flowchart of a method for screening difficult samples according to an embodiment of the present invention. The method can include the following steps:
  • S101 Use a pre-established target detection model to detect each obtained image to be screened, and determine the image to be screened that includes at least one first missed target region image.
  • the target detection model is: used to detect the area of the target contained in the image and determine the confidence that the detected target exists in the area of the target, and the first missed target area image is: the corresponding confidence is lower than the preset threshold image.
  • the target detection model is: a network model trained based on an image marked with a target to be detected.
  • the method can be applied to any type of electronic device with computing capability, and the electronic device can be a server or a terminal device.
  • the pre-established target detection model can be a neural network model, for example: it can be a convolutional neural network model, specifically it can be Faster R-CNN (Faster Region-Convolutional Neural Networks, fast region-convolution) Neural network model) and YOLO (You Only Look Once) model.
  • the pre-established target detection model can be any type of neural network model that can detect the location of the target in the image in related technologies. The specific types of pre-established target detection models are defined. For the training method of the pre-established target detection model, reference may be made to related technologies, and the embodiment of the present invention does not specifically limit it.
  • the target to be detected may be any type of target, including but not limited to lane lines, vehicles, traffic lights, signs, and/or pedestrians.
  • the electronic device after the electronic device obtains one or more frames of images to be screened, it can use a pre-established target detection model to detect each of the obtained images to be screened, and separate the regions where the target may exist in the image to be screened. Identify and determine the respective confidence level of each area where the target may exist; cut out the area image corresponding to the identified area where the target may exist, and subsequently, based on the corresponding confidence level of each area image, The region image whose corresponding confidence is lower than the preset threshold is determined from the intercepted region image as the first missed detection target region image, and it can be determined from the obtained images to be screened that at least one first missed detection is included The image to be filtered of the target area image.
  • the confidence level can represent the possibility that the corresponding regional image has the target to be detected.
  • the lower the confidence of the region image is, the less likely it is that the target detection model predicts that there is a target to be detected in the region image of the image to be screened.
  • the confidence level corresponding to the region image is low, there is a possibility that the region image contains the missed target to be detected.
  • the intercepted region image corresponding to the region where the target to be detected may exist may include the region image whose corresponding confidence is within a preset confidence threshold, and the lower limit of the preset confidence threshold is 0, The upper limit is greater than or equal to the aforementioned preset threshold.
  • the electronic device may mark and record the correspondence between the to-be-screened image and the at least one first missed-detected target area image contained in it. To be used in subsequent processes.
  • S102 Perform image feature extraction on each first missed target area image, and determine the image feature of each first missed target area image.
  • the electronic device can use any type of preset feature extraction algorithm to perform image feature extraction on each first missed target area image to determine the image feature of each first missed target area image.
  • the preset feature extraction algorithm may include, but is not limited to, SIFT (Scale-invariant feature transform) feature extraction algorithm, HOG (Histogram of Oriented Gradient, directional gradient histogram) feature extraction algorithm, Harr feature extraction Algorithm and GIST (Global Feature) extraction algorithm, etc.
  • the preset feature extraction algorithm can also be a feature extraction algorithm of convolutional neural network.
  • S103 Determine the target label corresponding to each first missed target area image based on the image feature of each first missed target area image and the pre-established corresponding relationship.
  • the correspondence relationship includes: the correspondence relationship between the image features of the annotated image and their corresponding tags.
  • the pre-established correspondence relationship may be stored locally or in a storage device connected to the electronic device, and the correspondence relationship includes: the correspondence relationship between the image features of the marked image and the corresponding label, and the marked image may include
  • the pre-established target detection model cuts out from the original image in which the corresponding confidence is lower than the preset threshold region image, the original image may refer to the later-mentioned established image; the marked image may also include: The captured image contains or does not contain the target to be detected. In this case, correspondingly, in order to ensure the accuracy of the difficult sample screening process, the marked image may include only the target to be detected, or only the target that is not to be detected. .
  • the label corresponding to each labeled image may include: a missed detection label that characterizes that the labeled image contains a target to be detected that is missed by the pre-established target detection model, or that the labeled image does not contain a target that is pre-established
  • the non-missed label of the target to be detected that is missed by the detection model for example: the label corresponding to the marked image may be a label that characterizes that the marked image includes a lane line, that is, a missed label, that is, the content of the missed label can be: "Lane line", or a label indicating that the lane line is not included in the marked image, that is, a non-missing label, and the content of the non-missing label may be "non-lane line".
  • the pre-established correspondence relationship may be stored in a preset index database, so as to compare and match the image feature of the first missed-detected target area image with the image feature of the marked image in the correspondence relationship.
  • the electronic device may match the image feature of the first missed target area image with the image feature of each labeled image in the corresponding relationship for each first missed target area, and the corresponding relationship is with The label corresponding to the image feature that most matches the image feature of the first missed target area image is determined to be the target label corresponding to the first missed target area image.
  • the above-mentioned matching process may be: based on a preset similarity algorithm, calculating the similarity value between the image feature of the first missed-detected target area image and the image feature of each labeled image in the corresponding relationship, correspondingly,
  • the image feature that best matches the image feature of the first missed target area image in the above correspondence may refer to the image feature with the largest similarity value between the image feature of the first missed target area image in the corresponding relationship.
  • the preset similarity algorithms include but are not limited to: Euclidean distance, cosine distance, Min-type distance, correlation coefficient and other algorithms.
  • the S103 may include the following steps 01-02:
  • the electronic device may, for each first missed target area image, based on the image feature of the first missed target area image and the image feature of the labeled image in the pre-established correspondence, from the corresponding relationship , It is determined that a plurality of labeled images that match the image feature of the first missed target area image is determined, and the labels corresponding to the multiple labeled images that match the image feature of the first missed target area image are determined as the The candidate label corresponding to the first missed target area image, and further, based on the candidate label corresponding to each first missed target area image, the target label corresponding to each first missed target area image is determined. In order to improve the accuracy of the target label corresponding to each first missed target area image determined to a certain extent.
  • the 01 may include the following steps 011-012:
  • 012 Determine the candidate label corresponding to each first missed target area image based on the similarity value.
  • the electronic device calculates the image based on the preset similarity algorithm, the image features of the first missed target area image, and the image features of each labeled image.
  • the similarity value between the first missed target area image and each labeled image, and then, based on the similarity value, the candidate label corresponding to each first missed target area image is determined, for example: the corresponding similarity
  • the preset number of labels corresponding to the marked image with the largest value is determined as the candidate label corresponding to the first missed target area image.
  • the 012 may include the following steps:
  • the first preset number of labels in the label queue corresponding to the first missed target area image are determined as candidate labels corresponding to the suspected target area image.
  • the electronic device may for each first missed target area image, according to the first missed target area image and The similarity value between each marked image is in descending order, and the label corresponding to each marked image is arranged to obtain the label queue corresponding to the first missed target area image; further, from the first missed detection The first preset number of tags in the tag queue corresponding to the target area image are determined as candidate tags corresponding to the suspected target area image.
  • it can be: for each first missed-detected target area image, according to the order of the similarity value between the first missed-detected target area image and each marked image, arrange the corresponding to each marked image To obtain another label queue corresponding to the first missed target area image; further, a preset number of labels from another label queue corresponding to the first missed target area image are determined to be the suspect target area
  • the candidate label corresponding to the image is also possible.
  • the preset number is a preset number, and it can also be set independently by the electronic device according to the number of image features of the marked image contained in the pre-established correspondence relationship, which is all right.
  • the 02 may include the following steps 021-224:
  • each candidate label corresponding to the image of the first missed detection target area may include a missed detection label and/or a non-missed detection label.
  • the electronic device may, for each first missed target area image, count the number of candidate labels corresponding to the first missed target area image, which is the missed label, as the first quantity ; And determine whether the first number meets the preset statistical conditions, that is, whether the first number is greater than the preset number threshold, or whether the ratio of the first number to the total number of candidate labels corresponding to the first missed target area image is determined Greater than the preset ratio threshold.
  • the first number is greater than the preset number threshold, or it is determined that the ratio of the first number to the total number of candidate labels corresponding to the first missed target area image is greater than the preset ratio threshold, then it is determined that the first number meets the preset threshold. Assuming the statistical condition, that is, among the candidate labels corresponding to the first missed target area image, the proportion of labels representing the first missed target area image containing the missed target to be detected is relatively large, and accordingly, the first missed target area image can be determined.
  • the target label corresponding to the detected target area image is a missed label; on the contrary, if it is determined that the first number is not greater than the preset number threshold, or it is determined that the first number corresponds to the total number of candidate labels corresponding to the first missed target area image If the ratio is not greater than the preset ratio threshold, it is determined that the first number does not meet the preset statistical conditions, that is, among the candidate labels corresponding to the first missed target area image, the first missed target area image contains the missed target to be detected The proportion of the labels of is small, and accordingly, it can be determined that the target label corresponding to the first missed target area image is a non-missed label.
  • the electronic device can set a weight value when determining the target label corresponding to each first missed target area image based on the candidate label corresponding to each first missed target area image, where the image of the marked image The greater the similarity value between the feature and the image feature of the first missed-detected target region image, the greater the weight value corresponding to the label corresponding to the labeled image.
  • the corresponding target label is a non-missing label. Wherein, it may be: the value corresponding to the candidate label of the missing label is 1, and the value corresponding to the candidate label of the non-missing label is 0.
  • S104 Determine an image to be screened that includes at least one first missed target area image whose corresponding target label is a missed label as a difficult sample image.
  • each image to be screened may not include the first missing-detected target area image, or may include at least one first missing-detected target area image.
  • the to-be-screened image includes at least one first missing-detected target area image, then It can be considered that the image to be screened contains the target to be detected that is missed by the pre-established target detection model, and the electronic device can determine the image to be screened as a difficult sample.
  • the difficult sample can be re-stored and annotated, and the corresponding relationship between each first missed target area image and its corresponding image to be screened can be saved. Furthermore, using the difficult sample and its labeling information to continue training the pre-established target detection model, that is, using the difficult sample and its labeling information to update the parameters of the pre-established target detection model, so as to improve the performance of the pre-established target detection model. Detection accuracy.
  • the first missed target area image of the label contains the missed target, and the image to be screened that contains at least one first missed target area image whose corresponding target label is the missed label is determined as a difficult sample image.
  • the image of the missed target area is extracted.
  • the content of the partial image block of the screened image is used as the main body of the search to avoid the interference of irrelevant information, effectively improve the accuracy of the search and recognition, and save the memory and increase the speed, so as to realize the automatic screening of difficult samples.
  • the S104 may include the following steps 11-14:
  • the candidate target area is a rectangular area
  • the area image corresponding to the candidate target area of the rectangular area will be determined as the first missed target area image
  • the candidate target area is a non-rectangular area, determine the area image corresponding to the smallest rectangular area containing the candidate target area as the first missed target area image to determine that it contains at least one first missed target The image to be filtered of the area image.
  • the electronic device may use a pre-established target detection model to detect each image to be screened, determine the image to be screened containing at least one suspected target area, and determine the confidence level corresponding to each suspected target area.
  • the at least one suspected target area is the aforementioned area that may include the target to be detected, and the image block represented by each suspected target area may be referred to as a regional image.
  • the suspected target area determined by the electronic device is an area whose corresponding confidence is within a preset confidence threshold.
  • the upper limit of the preset reliability threshold may not be less than the preset threshold, and the lower limit may be 0.
  • the electronic device determines a suspected target area whose corresponding confidence is lower than a preset threshold from the suspected target area as a candidate target area. And determine whether each candidate target area is a rectangle. If the candidate target area is a rectangular area, the area image corresponding to the candidate target area of the rectangular area will be determined as the first missed target area image; if the candidate target area is The area is a non-rectangular area, and the area image corresponding to the smallest rectangular area containing the candidate target area is determined as the first missed target area image to determine the image to be screened containing at least one first missed target area image .
  • the method may further include:
  • the process of establishing the corresponding relationship may include:
  • S201 Obtain established images and annotation information corresponding to each established image.
  • the labeling information includes: labeling location information of the area where the target contained in the corresponding established image is located.
  • S202 Use the target detection model to detect each established image, and determine each established image including at least one second missed target area image and detection position information corresponding to the at least one second missed target area image.
  • At least one second missed-detected target area image is an area image with a corresponding confidence level lower than a preset threshold
  • S203 Perform image feature extraction on each second missed target area image, and determine the image feature of each second missed target area image.
  • the electronic device may also include a process of establishing a corresponding relationship.
  • the electronic device can obtain multiple images for establishing the corresponding relationship.
  • the embodiment of the present invention is called an established image.
  • the established image containing the target to be detected can be marked with the area where the target to be detected is located, and the established image containing the target to be detected can be marked.
  • the corresponding labeling information contains the position information of the target to be detected in the corresponding established image, which can be referred to as labeling position information.
  • Input the established image and its corresponding annotation information into the pre-established target detection model use the pre-established target detection model to detect each established image, and determine each established image that contains at least one second missed target area image and Detection location information corresponding to at least one second missed-detected target area image; wherein each second missed-detected target area image is an image of an area with a corresponding confidence level lower than a preset threshold.
  • the electronic device After the electronic device obtains the at least one second missed target area image, it uses a preset feature extraction algorithm to perform image feature extraction on the at least one second missed target area image to obtain the image feature of the at least one second missed target area image; And, for each second missed target area image, based on the detection location information corresponding to the second missed target area image, and the annotation location information in the annotation information corresponding to the established image where the second missed target area image is located, Determine the label corresponding to the second missed target area image.
  • the S204 may be: determining the detection frame corresponding to the detection position information corresponding to the second missed target area image, and the label in the annotation information corresponding to the established image where the second missed target area image is located Whether the overlapping area between the label boxes corresponding to the location information exceeds the preset area ratio, if it exceeds the preset area ratio, it can be considered that there is a pending target detection model missed by the pre-established target detection model in the second missed target area image Detect the target and determine that the label corresponding to the second missed target area image is the missed label; on the contrary, if it does not exceed the preset area ratio, it can be considered that there is no pre-established target detection model in the second missed target area image For the missed target to be detected, it is determined that the label corresponding to the image of the second missed target area is a non-missed label.
  • the S204 may include the following steps 021-224:
  • intersection ratio corresponding to the second missed target area image is not less than the preset intersection ratio threshold, determine that the label corresponding to the second missed target area image is the missed label
  • intersection ratio corresponding to the second missed target area image is less than the preset intersection ratio threshold, determine that the label corresponding to the second missed target area image is a non-missed label to establish a corresponding relationship.
  • the electronic device may, for each second missed target area image, be based on the detection position information corresponding to the second missed target area image, and the annotation information corresponding to the established image where the second missed target area image is located.
  • the label position information in the image to determine the intersection of the label frame and the detection frame corresponding to the second missed target area image and the ratio between the unions, that is, the intersection and union ratio, where the label frame corresponds to the second missed target area
  • the marked location information of the image, and the detection frame corresponds to the detection location information corresponding to the second missed-detected target area image.
  • the label corresponding to the second missed target area image is determined to be a missed label; otherwise, if it is judged to be less than, it is considered that there is no pre-established target detection model in the second missed target area image
  • the label corresponding to the second missed target area image is determined to be a non-missed label, and the corresponding relationship between the image characteristics of each second missed target area image and the corresponding label is recorded, In order to establish the corresponding relationship between the image features of the marked image and the corresponding label.
  • the labeled image includes the above-mentioned second miss-detected target area image.
  • an embodiment of the present invention provides a difficult sample screening device, as shown in FIG. 3, which may include:
  • the first determining module 310 is configured to use a pre-established target detection model to detect each obtained image to be screened, and determine the image to be screened containing at least one first missed target area image, wherein the target
  • the detection model is: used to detect the area of the target contained in the image and determine the confidence of the existence of the target in the area where the detected target is located, and the first missed target area image is: an image of an area with a corresponding confidence lower than a preset threshold ;
  • the second determining module 320 is configured to perform image feature extraction on each first missed target area image, and determine the image feature of each first missed target area image;
  • the third determining module 330 is configured to determine the target label corresponding to each first missed target area image based on the image feature of each first missed target area image and the corresponding relationship established in advance, wherein the corresponding relationship Including: the correspondence between the image features of the annotated images and their corresponding labels;
  • the fourth determining module 340 is configured to determine the image to be screened containing at least one first missed target area image corresponding to the missed label as the difficult sample image.
  • the first missed target area image of the label contains the missed target, and the image to be screened that contains at least one first missed target area image whose corresponding target label is the missed label is determined as a difficult sample image.
  • the image of the missed target area is extracted.
  • the content of the partial image block of the screened image is used as the main body of the search to avoid the interference of irrelevant information, effectively improve the accuracy of the search and recognition, and save the memory and increase the speed, so as to realize the automatic screening of difficult samples.
  • the third determining module 330 includes:
  • the first determining unit is configured to determine the candidate label corresponding to each first missed target area image based on the image feature of each first missed target area image and the corresponding relationship established in advance;
  • the second determining unit is configured to determine the target label corresponding to each first missed target area image based on the candidate label corresponding to each first missed target area image.
  • the first determining unit includes:
  • the first determining sub-module is configured to determine the first missed target based on the image feature of the first missed target area image and the image feature of each marked image for each first missed target area image The similarity value between the regional image and each marked image;
  • the second determining sub-module is configured to determine, based on the similarity value, a candidate label corresponding to each first missed target area image.
  • the second determining submodule is specifically configured to, for each first missed target area image, according to the difference between the first missed target area image and each marked image Arrange the labels corresponding to each marked image in descending order of the similarity value to obtain the label queue corresponding to the first missed-detected target area image;
  • the first preset number of labels in the label queue corresponding to the first missed target area image are determined as candidate labels corresponding to the suspected target area image.
  • the second determining unit is specifically configured to, for each first missed-detected target area image, count the candidate labels corresponding to the first missed-detected target area image.
  • the number of candidate labels for the inspection label shall be regarded as the first quantity;
  • the meeting the preset statistical condition includes: greater than a preset number threshold, or a ratio of the total number of candidate tags corresponding to the corresponding first missed target area image Greater than the preset ratio threshold;
  • the target label corresponding to the first missed target area image is a non-missed label.
  • the first determining module 310 is specifically configured to use a pre-established target detection model to detect each obtained image to be screened, determine the image to be screened containing at least one suspected target area, and determine The confidence level corresponding to each suspected target area;
  • the area image corresponding to the candidate target area of the rectangular area will be determined as the first missed target area image
  • the area image corresponding to the smallest rectangular area containing the candidate target area is determined as the first missed target area image to determine the image that contains at least one first missed target area The image to be filtered.
  • the device further includes:
  • the relationship establishment module is configured to establish a corresponding relationship before determining the candidate label corresponding to each first missed target area image based on the image characteristics of each first missed target area image and the pre-established corresponding relationship
  • the process of, wherein the relationship establishment module includes:
  • the obtaining unit is configured to obtain the establishment image and the annotation information corresponding to each establishment image, wherein the annotation information includes: the annotation location information of the area where the target contained in the corresponding establishment image is located;
  • the third determining unit is configured to use the target detection model to detect each established image, and determine each established image including at least one second missed target area image and the at least one second missed target area Detection position information corresponding to the image, wherein the at least one second missed-detected target area image is an area image with a corresponding confidence level lower than the preset threshold;
  • the fourth determining unit is configured to perform image feature extraction on each second missed target area image, and determine the image feature of each second missed target area image;
  • the fifth determining unit is configured to, for each second missed target area image, based on the detection position information corresponding to the second missed target area image, and the annotation information corresponding to the established image where the second missed target area image is located To determine the label corresponding to the second miss-detected target area image in order to establish and obtain the corresponding relationship.
  • the fifth determining unit is specifically configured to, for each second missed target area image, based on the detection position information corresponding to the second missed target area image, and the second missed target area image. 2.
  • the label location information in the label information corresponding to the established image where the image of the missed target area is located, and the intersection ratio between the label frame and the detection frame corresponding to the second missed target area image is determined;
  • intersection ratio corresponding to the second missed target area image is not less than the preset intersection ratio threshold, determine that the label corresponding to the second missed target area image is the missed label
  • intersection ratio corresponding to the second missed target area image is less than the preset intersection ratio threshold, the label corresponding to the second missed target area image is determined to be a non-missed label to establish the corresponding relationship.
  • modules in the device in the embodiment may be distributed in the device in the embodiment according to the description of the embodiment, or may be located in one or more devices different from this embodiment with corresponding changes.
  • the modules of the above-mentioned embodiments can be combined into one module, or further divided into multiple sub-modules.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)

Abstract

本发明实施例公开一种困难样本筛选方法及装置，该方法包括：利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度；提取每一第一漏检目标区域图像的图像特征；基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，对应关系包括已标注图像的图像特征及其对应标签之间的对应关系；将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像，以实现自动化地筛选出困难样本。

Description

一种困难样本筛选方法及装置 技术领域
本发明涉及智能驾驶技术领域，具体而言，涉及一种困难样本筛选方法及装置。
背景技术
深度学习依赖于大量的训练数据即样本，但是当样本的数量达到一定规模后，不同的新增样本图像对模型性能提升的潜力是不一样的。
对于目标检测(ObjectDetection，也称物体检测)模型而言，困难样本即包含漏检目标和误检目标的样本，都是对提升目标检测模型的性能很有价值的数据。为了在一定程度上提升目标检测模型的性能，需要尽可能优先获取作为困难样本的样本，以利用困难样本训练优化相应的目标检测模型。
那么，如何自动化地从样本中筛选出困难样本成为亟待解决的问题。
发明内容
本发明提供了一种困难样本筛选方法及装置，以实现自动化地筛选出困难样本。具体的技术方案如下：
第一方面，本发明实施例提供了一种困难样本筛选方法，包括：
利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，其中，所述目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，所述第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像；
对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征；
基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，所述对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系；
将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。
可选的，所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签的步骤，包括：
基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签；
基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。
可选的，所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签的步骤，包括：
针对每一第一漏检目标区域图像，基于该第一漏检目标区域图像的图像特征，以及每一已标注图像的图像特征，确定该第一漏检目标区域图像与每一已标注图像之间的相似度值；
基于所述相似度值，确定每一第一漏检目标区域图像对应的备选标签。
可选的，所述基于所述相似度值，确定每一第一漏检目标区域图像对应的备选标签的步骤，包括：
针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从大到小的顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区域图像对应的标签队列；
针对每一第一漏检目标区域图像，将该第一漏检目标区域图像对应的标签队列中前预设数量个标签，确定为该疑似目标区域图像对应的备选标签。
可选的，所述基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签的步骤，包括：
针对每一第一漏检目标区域图像，统计该第一漏检目标区域图像对应的备选标签中，为漏检标签的备选标签的数量，作为第一数量；
判断所述第一数量是否满足预设统计条件，其中，所述满足预设统计条件包括：大于预设数量阈值，或与所对应第一漏检目标区域图像对应的备选标签的总数的比值大于预设比例阈值；
若判断所述第一数量满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为所述漏检标签；
若判断所述第一数量不满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为非漏检标签。
可选的，所述利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像的步骤，包括：
利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个疑似目标区域的待筛选图像，并确定每一疑似目标区域对应的置信度；
基于每一疑似目标区域对应的置信度，从所述疑似目标区域中确定出所对应置信度低于所述预设阈值的疑似目标区域，作为备选目标区域；
若备选目标区域为矩形区域，将为矩形区域的该备选目标区域对应的区域图像，确定为第一漏检目标区域图像；
若备选目标区域为非矩形区域，将包含该备选目标区域的最小的矩形区域对应的区域图像，确定为第一漏检目标区域图像，以确定出包含至少一个第一漏检目标区域图像的待筛选图像。
可选的，在所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签的步骤之前，所述方法还包括：
建立对应关系的过程，其中，所述过程包括：
获得建立图像以及每一建立图像对应的标注信息，其中，所述标注信息包括：所对应建立图像包含的目标所在区域的标注位置信息；
利用所述目标检测模型，对每一建立图像进行检测，确定包含至少一个第二漏检目标区域图像的每一建立图像及所述至少一个第二漏检目标区域图像对应的检测位置信息，其中，所述至少一个第二漏检目标区域图像为所对应置信度低于所述预设阈值的区域图像；
对每一第二漏检目标区域图像进行图像特征提取，确定每一第二漏检目标区域图像的图像特征；
针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系。
可选的，所述针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系的步骤，包括：
针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标注框与检测框之间的交并比；
针对每一第二漏检目标区域图像，将该第二漏检目标区域图像对应的交并比与预设交并比阈值进行比较；
若第二漏检目标区域图像对应的交并比不小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为漏检标签；
若第二漏检目标区域图像对应的交并比小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为非漏检标签，以建立得到所述对应关系。
第二方面，本发明实施例提供了一种困难样本筛选装置，包括：
第一确定模块，被配置为利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，其中，所述目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，所述第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像；
第二确定模块，被配置为对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征；
第三确定模块，被配置为基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，所述对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系；
第四确定模块，被配置为将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。
可选的，所述第三确定模块，包括：
第一确定单元，被配置为基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签；
第二确定单元，被配置为基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。
可选的，所述第一确定单元，包括：
第一确定子模块，被配置为针对每一第一漏检目标区域图像，基于该第一漏检目标区域图像的图像特征，以及每一已标注图像的图像特征，确定该第一漏检目标区域图像与每一已标注图像之间的相似度值；
第二确定子模块，被配置为基于所述相似度值，确定每一第一漏检目标区域图像对应的备选标签。
可选的，所述第二确定子模块，被具体配置为针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从大到小的顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区域图像对应的标签队列；
针对每一第一漏检目标区域图像，将该第一漏检目标区域图像对应的标签队列中前预设数量个标签，确定为该疑似目标区域图像对应的备选标签。
可选的，所述第二确定单元，被具体配置为针对每一第一漏检目标区域图像，统计该第一漏检目标区域图像对应的备选标签中，为漏检标签的备选标签的数量，作为第一数量；
判断所述第一数量是否满足预设统计条件，其中，所述满足预设统计条件包括：大于预设数量阈值，或与所对应第一漏检目标区域图像对应的备选标签的总数的比值大于预设 比例阈值；
若判断所述第一数量满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为所述漏检标签；
若判断所述第一数量不满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为非漏检标签。
可选的，所述第一确定模块，被具体配置为利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个疑似目标区域的待筛选图像，并确定每一疑似目标区域对应的置信度；
基于每一疑似目标区域对应的置信度，从所述疑似目标区域中确定出所对应置信度低于所述预设阈值的疑似目标区域，作为备选目标区域；
若备选目标区域为矩形区域，将为矩形区域的该备选目标区域对应的区域图像，确定为第一漏检目标区域图像；
若备选目标区域为非矩形区域，将包含该备选目标区域的最小的矩形区域对应的区域图像，确定为第一漏检目标区域图像，以确定出包含至少一个第一漏检目标区域图像的待筛选图像。
可选的，所述装置还包括：
关系建立模块，被配置为在所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签之前，建立对应关系的过程，其中，所述关系建立模块，包括：
获得单元，被配置为获得建立图像以及每一建立图像对应的标注信息，其中，所述标注信息包括：所对应建立图像包含的目标所在区域的标注位置信息；
第三确定单元，被配置为利用所述目标检测模型，对每一建立图像进行检测，确定包含至少一个第二漏检目标区域图像的每一建立图像及所述至少一个第二漏检目标区域图像对应的检测位置信息，其中，所述至少一个第二漏检目标区域图像为所对应置信度低于所述预设阈值的区域图像；
第四确定单元，被配置为对每一第二漏检目标区域图像进行图像特征提取，确定每一第二漏检目标区域图像的图像特征；
第五确定单元，被配置为针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系。
可选的，所述第五确定单元，被具体配置为针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像 对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标注框与检测框之间的交并比；
针对每一第二漏检目标区域图像，将该第二漏检目标区域图像对应的交并比与预设交并比阈值进行比较；
若第二漏检目标区域图像对应的交并比不小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为漏检标签；
若第二漏检目标区域图像对应的交并比小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为非漏检标签，以建立得到所述对应关系。
由上述内容可知，本发明实施例提供的一种困难样本筛选方法及装置，可以利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，其中，目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像；对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征；基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，对应关系包括已标注图像的图像特征及其对应标签之间的对应关系；将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。
应用本发明实施例，可以基于包括已标注图像的图像特征及其对应标签之间的对应关系的预先建立的对应关系，以及从第一漏检目标区域图像提取的图像特征，从已标注图像中，确定出图像特征与第一漏检目标区域图像的图像特征相似的已标注图像，进而，基于图像特征与第一漏检目标区域图像的图像特征相似的已标注图像对应的标签，确定每一第一漏检目标区域图像对应的目标标签，其中，该目标标签中可以包括表征该第一漏检目标区域图像中包含漏检目标的漏检标签；进而，可以认为对应的目标标签为漏检标签的第一漏检目标区域图像中包含漏检目标，将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。以实现可以有针对性地选出需要的样本，尤其是当需要样本有集中特性而无关样本复杂多变的场景；根据目标检测模型检测出的漏检目标区域，提取漏检目标区域图像即待筛选图像的局部图像块的内容作为检索的主体，避免无关信息的干扰，有效提高检索识别的准确率，而且节约内存提升速度，以实现自动化地筛选出困难样本。当然，实施本发明的任一产品或方法并不一定需要同时达到以上所述的所有优点。
本发明实施例的创新点包括：
1、基于包括已标注图像的图像特征及其对应标签之间的对应关系的预先建立的对应 关系，以及从第一漏检目标区域图像提取的图像特征，从已标注图像中，确定出图像特征与第一漏检目标区域图像的图像特征相似的已标注图像，进而，基于图像特征与第一漏检目标区域图像的图像特征相似的已标注图像对应的标签，确定每一第一漏检目标区域图像对应的目标标签，其中，该目标标签中可以包括表征该第一漏检目标区域图像中包含漏检目标的漏检标签；进而，可以认为对应的目标标签为漏检标签的第一漏检目标区域图像中包含漏检目标，将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。以实现可以有针对性地选出需要的样本，尤其是当需要样本有集中特性而无关样本复杂多变的场景；根据目标检测模型检测出的漏检目标区域，提取漏检目标区域图像即待筛选图像的局部图像块的内容作为检索的主体，避免无关信息的干扰，有效提高检索识别的准确率，而且节约内存提升速度，以实现自动化地筛选出困难样本。
2、在确定每一第一漏检目标区域图像对应的目标标签时，可以首先确定第一漏检目标区域图像与每一已标注图像之间的相似度值，进而，针对每一第一漏检目标区域图像，基于其对应的相似度值，确定出与该第一漏检目标区域图像最相似的前预设数量个已标注图像对应的标签，将上述所确定出的标签，作为第一漏检目标区域图像对应的备选标签；基于第一漏检目标区域图像对应的备选标签，确定第一漏检目标区域图像对应的目标标签，以在一定程度上提高所确定的目标标签的准确性。
3、统计每一第一漏检目标区域图像对应的备选标签中为漏检标签的备选标签的第一数量，并将所对应第一数量满足预设统计条件的第一漏检目标区域图像对应的目标标签确定为漏检标签，将所对应第一数量不满足预设统计条件的第一漏检目标区域图像对应的目标标签确定为非漏检标签，以在一定程度上提高所确定的包含漏检目标的第一漏检目标区域图像的准确性性，进而，提高自动确定困难样本的准确性。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单介绍。显而易见地，下面描述中的附图仅仅是本发明的一些实施例。对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例提供的困难样本筛选方法的一种流程示意图；
图2为本发明实施例提供的建立对应关系的一种流程示意图；
图3为本发明实施例提供的困难样本筛选装置的一种结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整的描述。显然，所描述的实施例仅仅是本发明的一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有付出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
需要说明的是，本发明实施例及附图中的术语“包括”和“具有”以及它们的任何变形，意图在于覆盖不排他的包含。例如包含的一系列步骤或单元的过程、方法、***、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其他步骤或单元。
本发明提供了一种困难样本筛选方法及装置，以实现自动化地筛选出困难样本。下面对本发明实施例进行详细说明。
图1为本发明实施例提供的困难样本筛选方法的一种流程示意图。该方法可以包括以下步骤：
S101：利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像。
其中，目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像。该目标检测模型为：基于标注有待检测目标的图像训练所得的网络模型。
本发明实施例中，该方法可以应用于任一类型的具有计算能力的电子设备中，该电子设备可以为服务器或者终端设备。
一种情况中，该预先建立的目标检测模型可以为神经网络模型，例如：可以为卷积神经网络模型，具体的可以为Faster R-CNN(Faster Region-Convolutional Neural Networks，快速地区域-卷积神经网络模型)以及YOLO(You Only Look Once)模型，该预先建立的目标检测模型可以为：相关技术中任一类型的可以检测图像中目标所在位置的神经网络模型，本发明实施例并不对该预先建立的目标检测模型的具体类型进行限定。对于对该预先建立的目标检测模型的训练方式可以参见相关技术，本发明实施例并不作具体限定。
其中，该待检测目标可以为任一类型的目标，包括但不限于车道线、车辆、交通灯、指示牌和/或行人等等。
一种实现方式中，电子设备获得一帧或多帧待筛选图像之后，可以利用预先建立的目标检测模型对所获得的每一待筛选图像进行检测，将待筛选图像中可能存在目标的区域分别标识出，并确定出每一可能存在目标的区域各自对应的置信度；并将所标识出的可能存在目标的区域对应的区域图像截取出，后续的，基于每一区域图像对应的置信度，从截取出的区域图像中确定出所对应置信度低于预设阈值的区域图像，作为第一漏检目标区域图 像，并可以从所获得的待筛选图像中，确定出包含至少一个第一漏检目标区域图像的待筛选图像。
其中，置信度可以表征所对应的区域图像存在待检测目标的可能性。一种情况中，区域图像对应的置信度越低，表征目标检测模型预测待筛选图像的该区域图像中存在待检测目标的可能性越小。相应的，该区域图像所对应的置信度较低时，该区域图像存在包含被漏检的待检测目标的可能性。相应的，所截取出的可能存在待检测目标的区域对应的区域图像中，可以包括所对应置信度处于预设置信度阈值内的区域图像，该预设置信度阈值的下限值为0，上限值大于或等于上述预设阈值。
一种情况中，电子设备在确定出包含至少一个第一漏检目标区域图像的待筛选图像之后，可以标记并记录待筛选图像与其包含的至少一个第一漏检目标区域图像之间的对应关系，以用于后续的流程。
S102：对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征。
本步骤中，电子设备可以利用任一类型的预设特征提取算法，对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征。其中，该预设特征提取算法可以包括但不限于SIFT(Scale-invariant feature transform，尺度不变特征变换)特征提取算法、HOG(Histogram of Oriented Gradient，方向梯度直方图)特征提取算法、Harr特征提取算法以及GIST(全局特征)提取算法等，该预设特征提取算法也可以是卷积神经网络类的特征提取算法。
S103：基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签。
其中，该对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系。
电子设备本地或所连接的存储设备中，可以预先存储有该预先建立的对应关系，该对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系，该已标注图像可以包括基于该预先建立的目标检测模型，从其所在原图像中截取出的所对应置信度低于预设阈值的区域图像，该原图像可以指后续提到的建立图像；该已标注图像也可以包括：所采集的包含或不包含待检测目标的图像，此类情况下，相应的，为了保证困难样本筛选流程的准确性，该以标注图像中可以仅包括待检测目标，或仅包括非待检测目标。
每一已标注图像对应的标签可以包括：表征该已标注图像包含被该预先建立的目标检测模型漏检的待检测目标的漏检标签，或表征该已标注图像不包含被该预先建立的目标检测模型漏检的待检测目标的非漏检标签，例如：该已标注图像对应的标签可以为表征已标注图像中包括车道线的标签即漏检标签，即该漏检标签可以包括的内容为“车道线”，或 者为表征已标注图像中不包括车道线的标签即非漏检标签，该非漏检标签可以包括的内容为“非车道线”。
在一种情况中，该预先建立的对应关系可以存储于预先设置的索引数据库中，以便于第一漏检目标区域图像的图像特征与对应关系中的已标注图像的图像特征的对比匹配。
一种实现方式中，电子设备可以针对每一第一漏检目标区域，将该第一漏检目标区域图像的图像特征与对应关系中每一已标注图像的图像特征进行匹配，对应关系中与该第一漏检目标区域图像的图像特征最匹配的图像特征对应的标签，确定为该第一漏检目标区域图像对应的目标标签。其中，上述匹配过程可以是：基于预先设置的相似度算法，计算该第一漏检目标区域图像的图像特征与对应关系中每一已标注图像的图像特征之间的相似度值，相应的，上述对应关系中与该第一漏检目标区域图像的图像特征最匹配的图像特征可以指：对应关系中与该第一漏检目标区域图像的图像特征之间的相似度值最大的图像特征。其中，预先设置的相似度算法包括但不限于：欧几里德距离、余弦距离、闵式距离以及相关系数等算法。
在另一种实现方式中，所述S103，可以包括如下步骤01-02：
01：基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签；
02：基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。
本实现方式中，电子设备可以针对每一第一漏检目标区域图像，基于该第一漏检目标区域图像的图像特征，和预先建立的对应关系中的已标注图像的图像特征，从对应关系中，确定出与该第一漏检目标区域图像的图像特征匹配多个已标注图像，将与该第一漏检目标区域图像的图像特征匹配的多个已标注图像对应的标签，确定为该第一漏检目标区域图像对应的备选标签，进而，基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。以在一定程度上提高所确定每一第一漏检目标区域图像对应的目标标签的准确性。
在一种实现方式中，所述01，可以包括如下步骤011-012：
011：针对每一第一漏检目标区域图像，基于该第一漏检目标区域图像的图像特征，以及每一已标注图像的图像特征，确定该第一漏检目标区域图像与每一已标注图像之间的相似度值；
012：基于相似度值，确定每一第一漏检目标区域图像对应的备选标签。
本实现方式中，电子设备针对每一第一漏检目标区域图像，基于预先设置的相似度算法、该第一漏检目标区域图像的图像特征以及每一已标注图像的图像特征，计算得到该第 一漏检目标区域图像与每一已标注图像之间的相似度值，进而，基于相似度值，确定每一第一漏检目标区域图像对应的备选标签，例如：将所对应相似度值最大的预设数量个已标注图像对应的标签，确定为该第一漏检目标区域图像对应的备选标签。
一种实现方式中，所述012，可以包括如下步骤：
针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从大到小的顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区域图像对应的标签队列；
针对每一第一漏检目标区域图像，将该第一漏检目标区域图像对应的标签队列中前预设数量个标签，确定为该疑似目标区域图像对应的备选标签。
为了能够从对应关系中，确定出所对应相似度值最大的预设数量个已标注图像对应的标签，电子设备可以针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从大到小的顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区域图像对应的标签队列；进而，从该第一漏检目标区域图像对应的标签队列中前预设数量个标签，确定为该疑似目标区域图像对应的备选标签。或者，可以是：针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从小到大的顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区域图像对应的另一标签队列；进而，从该第一漏检目标区域图像对应的另一标签队列中后预设数量个标签，确定为该疑似目标区域图像对应的备选标签，这也是可以的。
其中，该预设数量为预先设定的数量，也可以是电子设备根据预先建立的对应关系中包含的已标注图像的图像特征的数量自主设定的，这都是可以的。
在一种实现方式中，所述02，可以包括如下步骤021-024：
021：针对每一第一漏检目标区域图像，统计该第一漏检目标区域图像对应的备选标签中，为漏检标签的备选标签的数量，作为第一数量；
022：判断第一数量是否满足预设统计条件，其中，满足预设统计条件包括：大于预设数量阈值，或与所对应第一漏检目标区域图像对应的备选标签的总数的比值大于预设比例阈值；
023：若判断第一数量满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为漏检标签；
024：若判断第一数量不满足预设统计条件，确定该第一漏检目标区域图像对应的目标标签为非漏检标签。
其中，每一第一漏检目标区域图像对应的备选标签中，可以包括漏检标签和/或非漏检标签。本实施例中，电子设备可以针对每一第一漏检目标区域图像，统计该第一漏检目 标区域图像对应的备选标签中，为漏检标签的备选标签的数量，作为第一数量；并判断第一数量是否满足预设统计条件，即判断第一数量是否大于预设数量阈值，或判断第一数量与所对应第一漏检目标区域图像对应的备选标签的总数的比值是否大于预设比例阈值。
其中，若判断第一数量大于预设数量阈值，或判断第一数量与所对应第一漏检目标区域图像对应的备选标签的总数的比值大于预设比例阈值，则确定第一数量满足预设统计条件，即第一漏检目标区域图像对应的备选标签中，表征第一漏检目标区域图像包含漏检的待检测目标的标签的比例较大，相应的，则可以确定第一漏检目标区域图像对应的目标标签为漏检标签；反之，若判断第一数量不大于预设数量阈值，或判断第一数量与所对应第一漏检目标区域图像对应的备选标签的总数的比值不大于预设比例阈值，则确定第一数量不满足预设统计条件，即第一漏检目标区域图像对应的备选标签中，表征第一漏检目标区域图像包含漏检的待检测目标的标签的比例较小，相应的，则可以确定第一漏检目标区域图像对应的目标标签为非漏检标签。
在另一种实现方式中，考虑到已标注图像的图像特征与第一漏检目标区域图像的图像特征之间的相似度值越大，表征第一漏检目标区域图像与已标注图像越相似，并且，对于漏检的待检测目标，其之间的特征也是非常相似的。鉴于此，电子设备在基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签时，可以设置权重值，其中，已标注图像的图像特征与第一漏检目标区域图像的图像特征之间的相似度值越大，该已标注图像对应的标签对应的权重值越大。
后续的，可以针对每一第一漏检目标区域图像，将该第一漏检目标区域图像对应的每一备选标签对应的数值与其对应的权重值的乘积之和即第一和，与预设标签阈值进行比较，确定第一和大于预设标签阈值的第一漏检目标区域图像对应的目标标签为漏检标签，确定第一和不大于预设标签阈值的第一漏检目标区域图像对应的目标标签为非漏检标签。其中，可以是：为漏检标签的备选标签对应的数值为1，为非漏检标签的备选标签对应的数值为0。
S104：将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。
其中，每一待筛选图像中可以不包含第一漏检目标区域图像，或者可以包含至少一个第一漏检目标区域图像，当待筛选图像中包含至少一个第一漏检目标区域图像时，则，可以认为该待筛选图像中包含被预先建立的目标检测模型漏检的待检测目标，电子设备可以将该待筛选图像确定为困难样本。
在一种实现方式中，在确定出困难样本之后，可以对该困难样本重新进行存储以及标注，并保存每一第一漏检目标区域图像与其对应的待筛选图像之间的对应关系。进而，利 用该困难样本及其标注信息继续训练该预先建立的目标检测模型，即利用该困难样本及其标注信息更新该预先建立的目标检测模型的参数，以提高该预先建立的目标检测模型的检测精度。
应用本发明实施例，可以基于包括已标注图像的图像特征及其对应标签之间的对应关系的预先建立的对应关系，以及从第一漏检目标区域图像提取的图像特征，从已标注图像中，确定出图像特征与第一漏检目标区域图像的图像特征相似的已标注图像，进而，基于图像特征与第一漏检目标区域图像的图像特征相似的已标注图像对应的标签，确定每一第一漏检目标区域图像对应的目标标签，其中，该目标标签中可以包括表征该第一漏检目标区域图像中包含漏检目标的漏检标签；进而，可以认为对应的目标标签为漏检标签的第一漏检目标区域图像中包含漏检目标，将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。以实现可以有针对性地选出需要的样本，尤其是当需要样本有集中特性而无关样本复杂多变的场景；根据目标检测模型检测出的漏检目标区域，提取漏检目标区域图像即待筛选图像的局部图像块的内容作为检索的主体，避免无关信息的干扰，有效提高检索识别的准确率，而且节约内存提升速度，以实现自动化地筛选出困难样本。
在本发明的另一实施例中，所述S104，可以包括如下步骤11-14：
11：利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个疑似目标区域的待筛选图像，并确定每一疑似目标区域对应的置信度；
12：基于每一疑似目标区域对应的置信度，从疑似目标区域中确定出所对应置信度低于预设阈值的疑似目标区域，作为备选目标区域；
13：若备选目标区域为矩形区域，将为矩形区域的该备选目标区域对应的区域图像，确定为第一漏检目标区域图像；
14：若备选目标区域为非矩形区域，将包含该备选目标区域的最小的矩形区域对应的区域图像，确定为第一漏检目标区域图像，以确定出包含至少一个第一漏检目标区域图像的待筛选图像。
本实施例中，电子设备可以利用预先建立的目标检测模型，对每一待筛选图像进行检测，确定包含至少一个疑似目标区域的待筛选图像，并确定每一疑似目标区域对应的置信度。其中，该至少一疑似目标区域即上述的可能包括待检测目标的区域，每一疑似目标区域所表征的图像块，可以称为区域图像。电子设备所确定出的疑似目标区域为所对应置信度处于预设置信度阈值内的区域。该预设置信度阈值的上限值可以不小于预设阈值，下限值可以为0。
电子设备从疑似目标区域中确定出所对应置信度低于预设阈值的疑似目标区域，作为 备选目标区域。并且判断每一备选目标区域是否为矩形，若备选目标区域为矩形区域，将为矩形区域的该备选目标区域对应的区域图像，确定为第一漏检目标区域图像；若备选目标区域为非矩形区域，将包含该备选目标区域的最小的矩形区域对应的区域图像，确定为第一漏检目标区域图像，以确定出包含至少一个第一漏检目标区域图像的待筛选图像。
在本发明的另一实施例中，在所述S103之前，所述方法还可以包括：
建立对应关系的过程，如图2所示，所述过程可以包括：
S201：获得建立图像以及每一建立图像对应的标注信息。
其中，标注信息包括：所对应建立图像包含的目标所在区域的标注位置信息。
S202：利用目标检测模型，对每一建立图像进行检测，确定包含至少一个第二漏检目标区域图像的每一建立图像及至少一个第二漏检目标区域图像对应的检测位置信息。
其中，至少一个第二漏检目标区域图像为所对应置信度低于预设阈值的区域图像
S203：对每一第二漏检目标区域图像进行图像特征提取，确定每一第二漏检目标区域图像的图像特征。
S204：针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系。
本实施例中，电子设备还可以包括建立对应关系的过程。相应的，电子设备可以获得多张用于建立对应关系的图像，本发明实施例称之为建立图像，包含待检测目标的建立图像中可以标注有待检测目标所在区域，包含待检测目标的建立图像对应的标注信息包含待检测目标在所对应建立图像中的位置信息，可以称之为标注位置信息。将建立图像及其对应的标注信息输入预先建立的目标检测模型，利用该预先建立的目标检测模型对每一建立图像进行检测，确定包含至少一个第二漏检目标区域图像的每一建立图像及至少一个第二漏检目标区域图像对应的检测位置信息；其中，每一第二漏检目标区域图像为所对应置信度低于预设阈值的区域图像。
电子设备获得至少一个第二漏检目标区域图像之后，利用预设特征提取算法，对至少一个第二漏检目标区域图像进行图像特征提取，得到至少一个第二漏检目标区域图像的图像特征；并且，针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签。
一种情况中，所述S204，可以为：确定该第二漏检目标区域图像对应的检测位置信息对应的检测框，与该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息对应的标注框之间重叠的面积，是否超过预设面积比例，若超过预设面积比例，则可 以认为该第二漏检目标区域图像中存在被预先建立的目标检测模型漏检的待检测目标，确定第二漏检目标区域图像对应的标签为漏检标签；反之，若未超过预设面积比例，则可以认为该第二漏检目标区域图像中不存在被预先建立的目标检测模型漏检的待检测目标，确定第二漏检目标区域图像对应的标签为非漏检标签。
另一种情况中，所述S204，可以包括如下步骤021-024：
021：针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标注框与检测框之间的交并比；
022：针对每一第二漏检目标区域图像，将该第二漏检目标区域图像对应的交并比与预设交并比阈值进行比较；
023：若第二漏检目标区域图像对应的交并比不小于预设交并比阈值，确定该第二漏检目标区域图像对应的标签为漏检标签；
024：若第二漏检目标区域图像对应的交并比小于预设交并比阈值，确定该第二漏检目标区域图像对应的标签为非漏检标签，以建立得到对应关系。
本实现方式中，电子设备可以针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标注框与检测框之间的交集以及并集之间比值，即交并比，其中，该标注框对应第二漏检目标区域图像的标注位置信息，检测框对应第二漏检目标区域图像对应的检测位置信息。判断该第二漏检目标区域图像对应的交并比是否不小于预设交并比阈值，若判断为不小于，则认为该第二漏检目标区域图像中存在被预先建立的目标检测模型漏检的待检测目标，确定该第二漏检目标区域图像对应的标签为漏检标签；反之，若判断为小于，则认为该第二漏检目标区域图像中不存在被预先建立的目标检测模型漏检的待检测目标，确定该第二漏检目标区域图像对应的标签为非漏检标签，记录每一该第二漏检目标区域图像的图像特征及其对应的标签之间的对应关系，以建立得到已标注图像的图像特征及其对应的标签的对应关系。该已标注图像包括上述第二漏检目标区域图像。
相应于上述方法实施例，本发明实施例提供了一种困难样本筛选装置，如图3所示，可以包括：
第一确定模块310，被配置为利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，其中，所述目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，所述第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像；
第二确定模块320，被配置为对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征；
第三确定模块330，被配置为基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，所述对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系；
第四确定模块340，被配置为将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。
应用本发明实施例，可以基于包括已标注图像的图像特征及其对应标签之间的对应关系的预先建立的对应关系，以及从第一漏检目标区域图像提取的图像特征，从已标注图像中，确定出图像特征与第一漏检目标区域图像的图像特征相似的已标注图像，进而，基于图像特征与第一漏检目标区域图像的图像特征相似的已标注图像对应的标签，确定每一第一漏检目标区域图像对应的目标标签，其中，该目标标签中可以包括表征该第一漏检目标区域图像中包含漏检目标的漏检标签；进而，可以认为对应的目标标签为漏检标签的第一漏检目标区域图像中包含漏检目标，将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。以实现可以有针对性地选出需要的样本，尤其是当需要样本有集中特性而无关样本复杂多变的场景；根据目标检测模型检测出的漏检目标区域，提取漏检目标区域图像即待筛选图像的局部图像块的内容作为检索的主体，避免无关信息的干扰，有效提高检索识别的准确率，而且节约内存提升速度，以实现自动化地筛选出困难样本。
在本发明的另一实施例中，所述第三确定模块330，包括：
第一确定单元，被配置为基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签；
第二确定单元，被配置为基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。
在本发明的另一实施例中，所述第一确定单元，包括：
第一确定子模块，被配置为针对每一第一漏检目标区域图像，基于该第一漏检目标区域图像的图像特征，以及每一已标注图像的图像特征，确定该第一漏检目标区域图像与每一已标注图像之间的相似度值；
第二确定子模块，被配置为基于所述相似度值，确定每一第一漏检目标区域图像对应的备选标签。
在本发明的另一实施例中，所述第二确定子模块，被具体配置为针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从大到小的 顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区域图像对应的标签队列；
针对每一第一漏检目标区域图像，将该第一漏检目标区域图像对应的标签队列中前预设数量个标签，确定为该疑似目标区域图像对应的备选标签。
在本发明的另一实施例中，所述第二确定单元，被具体配置为针对每一第一漏检目标区域图像，统计该第一漏检目标区域图像对应的备选标签中，为漏检标签的备选标签的数量，作为第一数量；
判断所述第一数量是否满足预设统计条件，其中，所述满足预设统计条件包括：大于预设数量阈值，或与所对应第一漏检目标区域图像对应的备选标签的总数的比值大于预设比例阈值；
若判断所述第一数量满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为所述漏检标签；
若判断所述第一数量不满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为非漏检标签。
可选的，所述第一确定模块310，被具体配置为利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个疑似目标区域的待筛选图像，并确定每一疑似目标区域对应的置信度；
基于每一疑似目标区域对应的置信度，从所述疑似目标区域中确定出所对应置信度低于所述预设阈值的疑似目标区域，作为备选目标区域；
若备选目标区域为矩形区域，将为矩形区域的该备选目标区域对应的区域图像，确定为第一漏检目标区域图像；
若备选目标区域为非矩形区域，将包含该备选目标区域的最小的矩形区域对应的区域图像，确定为第一漏检目标区域图像，以确定出包含至少一个第一漏检目标区域图像的待筛选图像。
在本发明的另一实施例中，所述装置还包括：
关系建立模块，被配置为在所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签之前，建立对应关系的过程，其中，所述关系建立模块，包括：
获得单元，被配置为获得建立图像以及每一建立图像对应的标注信息，其中，所述标注信息包括：所对应建立图像包含的目标所在区域的标注位置信息；
第三确定单元，被配置为利用所述目标检测模型，对每一建立图像进行检测，确定包含至少一个第二漏检目标区域图像的每一建立图像及所述至少一个第二漏检目标区域图像对应的检测位置信息，其中，所述至少一个第二漏检目标区域图像为所对应置信度低于 所述预设阈值的区域图像；
第四确定单元，被配置为对每一第二漏检目标区域图像进行图像特征提取，确定每一第二漏检目标区域图像的图像特征；
第五确定单元，被配置为针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系。
在本发明的另一实施例中，所述第五确定单元，被具体配置为针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标注框与检测框之间的交并比；
针对每一第二漏检目标区域图像，将该第二漏检目标区域图像对应的交并比与预设交并比阈值进行比较；
若第二漏检目标区域图像对应的交并比不小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为漏检标签；
若第二漏检目标区域图像对应的交并比小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为非漏检标签，以建立得到所述对应关系。
上述装置、***实施例与方法实施例相对应，与该方法实施例具有同样的技术效果，具体说明参见方法实施例。装置实施例是基于方法实施例得到的，具体的说明可以参见方法实施例部分，此处不再赘述。
本领域普通技术人员可以理解：附图只是一个实施例的示意图，附图中的模块或流程并不一定是实施本发明所必须的。
本领域普通技术人员可以理解：实施例中的装置中的模块可以按照实施例描述分布于实施例的装置中，也可以进行相应变化位于不同于本实施例的一个或多个装置中。上述实施例的模块可以合并为一个模块，也可以进一步拆分成多个子模块。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明实施例技术方案的精神和范围。

Claims (10)

  1. 一种困难样本筛选方法，其特征在于，包括：
    利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，其中，所述目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，所述第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像；
    对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征；
    基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，所述对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系；
    将包含至少一个所对应目标标签为漏检标签的第一漏检目标区域图像的待筛选图像，确定为困难样本图像。
  2. 如权利要求1所述的方法，其特征在于，所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签的步骤，包括：
    基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签；
    基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。
  3. 如权利要求2所述的方法，其特征在于，所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签的步骤，包括：
    针对每一第一漏检目标区域图像，基于该第一漏检目标区域图像的图像特征，以及每一已标注图像的图像特征，确定该第一漏检目标区域图像与每一已标注图像之间的相似度值；
    基于所述相似度值，确定每一第一漏检目标区域图像对应的备选标签。
  4. 如权利要求3所述的方法，其特征在于，所述基于所述相似度值，确定每一第一漏检目标区域图像对应的备选标签的步骤，包括：
    针对每一第一漏检目标区域图像，按照该第一漏检目标区域图像与每一已标注图像之间的相似度值从大到小的顺序，排列每一已标注图像对应的标签，得到该第一漏检目标区 域图像对应的标签队列；
    针对每一第一漏检目标区域图像，将该第一漏检目标区域图像对应的标签队列中前预设数量个标签，确定为该疑似目标区域图像对应的备选标签。
  5. 如权利要求2所述的方法，其特征在于，所述基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签的步骤，包括：
    针对每一第一漏检目标区域图像，统计该第一漏检目标区域图像对应的备选标签中，为漏检标签的备选标签的数量，作为第一数量；
    判断所述第一数量是否满足预设统计条件，其中，所述满足预设统计条件包括：大于预设数量阈值，或与所对应第一漏检目标区域图像对应的备选标签的总数的比值大于预设比例阈值；
    若判断所述第一数量满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为所述漏检标签；
    若判断所述第一数量不满足所述预设统计条件，确定该第一漏检目标区域图像对应的目标标签为非漏检标签。
  6. 如权利要求1-5任一项所述的方法，其特征在于，所述利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像的步骤，包括：
    利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个疑似目标区域的待筛选图像，并确定每一疑似目标区域对应的置信度；
    基于每一疑似目标区域对应的置信度，从所述疑似目标区域中确定出所对应置信度低于所述预设阈值的疑似目标区域，作为备选目标区域；
    若备选目标区域为矩形区域，将为矩形区域的该备选目标区域对应的区域图像，确定为第一漏检目标区域图像；
    若备选目标区域为非矩形区域，将包含该备选目标区域的最小的矩形区域对应的区域图像，确定为第一漏检目标区域图像，以确定出包含至少一个第一漏检目标区域图像的待筛选图像。
  7. 如权利要求1-6任一项所述的方法，其特征在于，在所述基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签的步骤之前，所述方法还包括：
    建立对应关系的过程，其中，所述过程包括：
    获得建立图像以及每一建立图像对应的标注信息，其中，所述标注信息包括：所对应建立图像包含的目标所在区域的标注位置信息；
    利用所述目标检测模型，对每一建立图像进行检测，确定包含至少一个第二漏检目标区域图像的每一建立图像及所述至少一个第二漏检目标区域图像对应的检测位置信息，其中，所述至少一个第二漏检目标区域图像为所对应置信度低于所述预设阈值的区域图像；
    对每一第二漏检目标区域图像进行图像特征提取，确定每一第二漏检目标区域图像的图像特征；
    针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系。
  8. 如权利要求7所述的方法，其特征在于，所述针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标签，以建立得到所述对应关系的步骤，包括：
    针对每一第二漏检目标区域图像，基于该第二漏检目标区域图像对应的检测位置信息，以及该第二漏检目标区域图像所在建立图像对应的标注信息中的标注位置信息，确定该第二漏检目标区域图像对应的标注框与检测框之间的交并比；
    针对每一第二漏检目标区域图像，将该第二漏检目标区域图像对应的交并比与预设交并比阈值进行比较；
    若第二漏检目标区域图像对应的交并比不小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为漏检标签；
    若第二漏检目标区域图像对应的交并比小于所述预设交并比阈值，确定该第二漏检目标区域图像对应的标签为非漏检标签，以建立得到所述对应关系。
  9. 一种困难样本筛选装置，其特征在于，所述装置包括：
    第一确定模块，被配置为利用预先建立的目标检测模型，对所获得的每一待筛选图像进行检测，确定包含至少一个第一漏检目标区域图像的待筛选图像，其中，所述目标检测模型为：用于检测图像所包含目标所在区域及确定检测出的目标所在区域存在目标的置信度，所述第一漏检目标区域图像为：所对应置信度低于预设阈值的区域图像；
    第二确定模块，被配置为对每一第一漏检目标区域图像进行图像特征提取，确定每一第一漏检目标区域图像的图像特征；
    第三确定模块，被配置为基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的目标标签，其中，所述对应关系包括：已标注图像的图像特征及其对应标签之间的对应关系；
    第四确定模块，被配置为将包含至少一个所对应目标标签为漏检标签的第一漏检目标 区域图像的待筛选图像，确定为困难样本图像。
  10. 如权利要求9所述的装置，其特征在于，所述第三确定模块，包括：
    第一确定单元，被配置为基于每一第一漏检目标区域图像的图像特征以及预先建立的对应关系，确定每一第一漏检目标区域图像对应的备选标签；
    第二确定单元，被配置为基于每一第一漏检目标区域图像对应的备选标签，确定每一第一漏检目标区域图像对应的目标标签。
PCT/CN2020/094109 2019-09-20 2020-06-03 一种困难样本筛选方法及装置 WO2021051887A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910890908.7 2019-09-20
CN201910890908.7A CN112541372B (zh) 2019-09-20 2019-09-20 一种困难样本筛选方法及装置

Publications (1)

Publication Number Publication Date
WO2021051887A1 true WO2021051887A1 (zh) 2021-03-25

Family

ID=74883929

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/094109 WO2021051887A1 (zh) 2019-09-20 2020-06-03 一种困难样本筛选方法及装置

Country Status (2)

Country Link
CN (1) CN112541372B (zh)
WO (1) WO2021051887A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113627413A (zh) * 2021-08-12 2021-11-09 杭州海康威视数字技术股份有限公司 数据标注方法、图像比对方法及装置
CN114445811A (zh) * 2022-01-30 2022-05-06 北京百度网讯科技有限公司 一种图像处理方法、装置及电子设备
CN117710756A (zh) * 2024-02-04 2024-03-15 成都数之联科技股份有限公司 一种目标检测及模型训练方法、装置、设备、介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115033731B (zh) * 2022-07-04 2023-07-18 小米汽车科技有限公司 图像检索方法、装置、电子设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109034190A (zh) * 2018-06-15 2018-12-18 广州深域信息科技有限公司 一种动态选择策略的主动样本挖掘的物体检测***及方法
CN110084113A (zh) * 2019-03-20 2019-08-02 阿里巴巴集团控股有限公司 活体检测方法、装置、***、服务器及可读存储介质
US20190279091A1 (en) * 2018-03-12 2019-09-12 Carnegie Mellon University Discriminative Cosine Embedding in Machine Learning

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107122378B (zh) * 2017-01-13 2021-03-16 北京星选科技有限公司 对象处理方法、装置及移动终端
CN109697449A (zh) * 2017-10-20 2019-04-30 杭州海康威视数字技术股份有限公司 一种目标检测方法、装置及电子设备
CN109871730A (zh) * 2017-12-05 2019-06-11 杭州海康威视数字技术股份有限公司 一种目标识别方法、装置及监控设备
CN108446707B (zh) * 2018-03-06 2020-11-24 北方工业大学 基于关键点筛选及dpm确认的遥感图像飞机检测方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190279091A1 (en) * 2018-03-12 2019-09-12 Carnegie Mellon University Discriminative Cosine Embedding in Machine Learning
CN109034190A (zh) * 2018-06-15 2018-12-18 广州深域信息科技有限公司 一种动态选择策略的主动样本挖掘的物体检测***及方法
CN110084113A (zh) * 2019-03-20 2019-08-02 阿里巴巴集团控股有限公司 活体检测方法、装置、***、服务器及可读存储介质

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113627413A (zh) * 2021-08-12 2021-11-09 杭州海康威视数字技术股份有限公司 数据标注方法、图像比对方法及装置
CN113627413B (zh) * 2021-08-12 2024-06-04 杭州海康威视数字技术股份有限公司 数据标注方法、图像比对方法及装置
CN114445811A (zh) * 2022-01-30 2022-05-06 北京百度网讯科技有限公司 一种图像处理方法、装置及电子设备
CN117710756A (zh) * 2024-02-04 2024-03-15 成都数之联科技股份有限公司 一种目标检测及模型训练方法、装置、设备、介质
CN117710756B (zh) * 2024-02-04 2024-04-26 成都数之联科技股份有限公司 一种目标检测及模型训练方法、装置、设备、介质

Also Published As

Publication number Publication date
CN112541372A (zh) 2021-03-23
CN112541372B (zh) 2023-03-28

Similar Documents

Publication Publication Date Title
WO2021051887A1 (zh) 一种困难样本筛选方法及装置
US11455805B2 (en) Method and apparatus for detecting parking space usage condition, electronic device, and storage medium
US8902053B2 (en) Method and system for lane departure warning
CN109919002B (zh) 黄色禁停线识别方法、装置、计算机设备及存储介质
CN105321350A (zh) ***检测方法及装置
US11721088B2 (en) Image translation for image recognition to compensate for source image regional differences
CN111191611A (zh) 基于深度学习的交通标志标号识别方法
CN111626275B (zh) 一种基于智能视频分析的异常停车检测方法
Choi et al. Multiple vehicles detection and tracking based on scale-invariant feature transform
CN110751619A (zh) 一种绝缘子缺陷检测方法
CN111898491A (zh) 一种车辆逆向行驶的识别方法、装置及电子设备
WO2022048572A1 (zh) 目标识别方法、装置和电子设备
CN115830399B (zh) 分类模型训练方法、装置、设备、存储介质和程序产品
CN109389105A (zh) 一种基于多任务的虹膜检测和视角分类方法
CN113095301A (zh) 占道经营监测方法、***与服务器
CN114926791A (zh) 一种路口车辆异常变道检测方法、装置、存储介质及电子设备
CN114429577A (zh) 一种基于高置信标注策略的旗帜检测方法及***及设备
CN114462469B (zh) 目标检测模型的训练方法、目标检测方法及相关装置
CN109684953B (zh) 基于目标检测和粒子滤波算法进行猪只跟踪的方法及装置
CN112581495A (zh) 图像处理方法、装置、设备及存储介质
CN112132892A (zh) 目标位置标注方法、装置及设备
CN113610770A (zh) 一种车牌识别方法和装置及设备
CN112149698A (zh) 一种困难样本数据的筛选方法及装置
CN116206326A (zh) 缺失检测模型的训练方法、导流区的缺失检测方法及装置
CA3012927A1 (en) Counting objects in images based on approximate locations

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20864611

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20864611

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20864611

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 27.09.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20864611

Country of ref document: EP

Kind code of ref document: A1