WO2023029065A1 - 数据集质量评估方法、装置、计算机设备及存储介质 - Google Patents
数据集质量评估方法、装置、计算机设备及存储介质 Download PDFInfo
- Publication number
- WO2023029065A1 WO2023029065A1 PCT/CN2021/117109 CN2021117109W WO2023029065A1 WO 2023029065 A1 WO2023029065 A1 WO 2023029065A1 CN 2021117109 W CN2021117109 W CN 2021117109W WO 2023029065 A1 WO2023029065 A1 WO 2023029065A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- evaluation
- evaluated
- structured
- dimension
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000003860 storage Methods 0.000 title claims abstract description 9
- 238000011156 evaluation Methods 0.000 claims abstract description 154
- 238000013441 quality evaluation Methods 0.000 claims abstract description 30
- 238000001303 quality assessment method Methods 0.000 claims description 43
- 238000012549 training Methods 0.000 claims description 30
- 230000007717 exclusion Effects 0.000 claims description 22
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 238000009826 distribution Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 18
- 238000013473 artificial intelligence Methods 0.000 description 15
- 238000011161 development Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Definitions
- the present invention relates to the field of information technology, in particular to a data set quality evaluation method, device, computer equipment and storage medium.
- Data is the basis for the development and application of artificial intelligence. Data sets are crucial to artificial intelligence algorithms. Using different quality data sets will result in different model parameters after training, resulting in different execution effects, which in turn affect the security of artificial intelligence algorithms. If criminals use attack methods to maliciously modify and add data sets, it will lead to model prediction errors. Therefore, how to effectively detect and evaluate the quality of data sets has become an urgent problem to be solved in artificial intelligence security.
- the invention provides a data set quality assessment method, device, computer equipment and storage medium, mainly aiming at improving the assessment accuracy and assessment efficiency of the data set quality.
- a data set quality assessment method including:
- a data set quality assessment device including:
- an acquisition unit configured to acquire the data to be evaluated in the data set
- a statistical unit configured to separately count attribute characteristics of the data to be evaluated under multiple evaluation dimensions
- the evaluation unit is configured to perform quality evaluation on the data to be evaluated based on attribute characteristics under the multiple evaluation dimensions, and obtain quality evaluation results of the data to be evaluated under the multiple evaluation dimensions.
- a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented:
- a computer device including a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the following steps when executing the program:
- a data set quality evaluation method, device, computer equipment and storage medium provided by the present invention compared with the current method of evaluating the quality of the data set by technicians based on their own experience, the present invention can obtain the waiting data in the data set Evaluate the data; and separately count the attribute characteristics of the data to be evaluated under multiple evaluation dimensions; at the same time, based on the attribute characteristics under the multiple evaluation dimensions, perform quality assessment on the data to be evaluated, and obtain the The quality evaluation results of the data to be evaluated under the multiple evaluation dimensions, so that by counting the attribute characteristics of the data to be evaluated under the multiple evaluation dimensions, the quality of the data set can be automatically evaluated from multiple evaluation dimensions, so that It can improve the evaluation accuracy and evaluation efficiency of the data set quality, and effectively guarantee the safety of the data set in the process of artificial intelligence development.
- FIG. 1 shows a flowchart of a data set quality assessment method provided by an embodiment of the present invention
- FIG. 2 shows a flowchart of another data set quality assessment method provided by an embodiment of the present invention
- Fig. 3 shows a schematic structural diagram of a data set quality assessment device provided by an embodiment of the present invention
- Fig. 4 shows a schematic structural diagram of another data set quality assessment device provided by an embodiment of the present invention.
- FIG. 5 shows a schematic diagram of a physical structure of a computer device provided by an embodiment of the present invention.
- an embodiment of the present invention provides a data set quality assessment method, as shown in Figure 1, the method includes:
- the data set includes a training data set and a prediction data set
- the data to be evaluated may specifically be each sample data in the training data set, or each prediction data in the prediction data set.
- the embodiment of the present invention develops a set of data set quality evaluation tools, which can The quality of the data set is automatically evaluated from multiple evaluation dimensions, thereby improving the evaluation accuracy and efficiency of the data set quality, and at the same time ensuring the safety of the data set in the process of artificial intelligence development.
- the embodiments of the present invention are mainly applied to scenarios where the quality of a data set is evaluated in multiple dimensions.
- the execution subject of the embodiment of the present invention is a device or device capable of evaluating the quality of a data set, which may be specifically set on the server side.
- the training data set and the prediction data set that need to be evaluated for quality are collected in advance, and the data in the training data set and the prediction data set may specifically be structured data or unstructured data, such as image data.
- the technician can click the file upload button on the data set quality evaluation tool interface to upload the training data set or prediction data set to be evaluated to the data set quality evaluation tool, so that The dataset quality assessment tool performs multi-dimensional quality assessment on the dataset to be assessed.
- the multiple assessment dimensions include data scale assessment dimension, data balance assessment dimension, data accuracy assessment dimension, data pollution assessment dimension and data bias assessment dimension. It should be noted that the assessment dimensions in the embodiments of the present invention are not limited to In addition to the evaluation dimensions listed above, other evaluation dimensions can also be included, which can be set according to actual business needs.
- the attribute characteristics of the data to be evaluated under the data size evaluation dimension include: the total amount of data, the number of features, the memory size occupied by the data, whether the data has labels, etc.; the attribute characteristics of the data to be evaluated under the evaluation dimension under data balance include The proportion of data volume under various tags; the attribute characteristics of the data to be evaluated under the data accuracy evaluation dimension include: the total amount of data, the total amount of missing tags, whether various tags are abnormal, etc.; the data to be evaluated under the data pollution evaluation dimension
- the attribute characteristics include: the amount of noise data, the amount of confrontation data, etc.; the attribute characteristics of the data to be evaluated under the data bias evaluation dimension include the bias characteristics corresponding to the data to be evaluated.
- different statistical methods can be used to separately count the attribute characteristics of the data to be evaluated in the data scale evaluation dimension, data balance evaluation dimension, data accuracy evaluation dimension, data pollution evaluation dimension and data bias evaluation dimension,
- the specific statistical methods adopted for different evaluation dimensions are different, see steps 202-206 for details.
- the evaluation standards corresponding to different evaluation dimensions are different.
- the data to be evaluated does not meet the evaluation standard corresponding to any evaluation dimension, If it is determined that the data to be evaluated has quality problems, the data set cannot be used to train or predict the model, and technicians need to re-collect the data set or perform data cleaning on the data set with quality problems.
- the “Yes” label corresponds to 90% of the data volume, and the “No” label corresponds to 10%.
- evaluation data can also be treated from the dimensions of data scale evaluation, data accuracy evaluation, data pollution evaluation, and data bias evaluation Carry out quality assessment. If the data to be evaluated does not meet the evaluation standards corresponding to the above dimensions, it is determined that the data to be evaluated has quality problems, and it cannot be used to train or predict the model. For the quality evaluation process of different evaluation dimensions, see steps 202-206. .
- the method for assessing the quality of a data set provided by the embodiment of the present invention, compared with the current method of assessing the quality of the data set by technicians based on their own experience, this method can obtain the data to be evaluated in the data set; and count them separately The attribute characteristics of the data to be evaluated under multiple evaluation dimensions; at the same time, based on the attribute characteristics under the multiple evaluation dimensions, the quality of the data to be evaluated is evaluated, and the data to be evaluated are respectively obtained.
- the quality evaluation results under multiple evaluation dimensions are described, so by counting the attribute characteristics of the data to be evaluated under multiple evaluation dimensions, the quality of the data set can be automatically evaluated from multiple evaluation dimensions, so that the quality of the data set can be improved. Evaluation accuracy and evaluation efficiency effectively guarantee the safety of data sets in the process of artificial intelligence development.
- the embodiment of the present invention provides another method for detecting contaminated sample data, as shown in FIG. 2 , the The methods described include:
- the data set includes a training data set and a prediction data set
- the data to be evaluated may specifically be each sample data in the training data set, or each prediction data in the prediction data set.
- the specific method of obtaining the data set is exactly the same as that of step 101, and will not be repeated here.
- step 202 specifically includes: according to the structured data and its corresponding label category, using a preset interpolation algorithm to simulate Combining the function curve corresponding to the structured data; using the function curve to predict the structured data to obtain the predicted label category corresponding to the structured data; based on the corresponding predicted label category and the structured data label category to determine whether the structured data is noise data.
- the preset interpolation algorithm performs interpolation processing on the structured data.
- the preset interpolation algorithm may specifically be a preset Kriging interpolation algorithm.
- the classification result may specifically be a classification probability value corresponding to the structured data.
- the structured data of the known classification results are x1, x2, x3, and the structured data to be interpolated is determined to be x4. Since the classification result corresponding to the structural data x4 to be interpolated is unknown, the known classification results can be used Structured data x1, x2, x3, estimate the classification result corresponding to the structured data x4 to be interpolated, specifically, calculate the difference between the structured data x4 to be interpolated and the structured data x1, x2, x3 of known classification results The larger the distance, the farther the structured data of the known classification result is from the structured data to be interpolated, and the impact on the structured data to be interpolated is smaller, so the corresponding data weight is smaller; On the contrary, the smaller the distance, the closer the structured data of the known classification result is to the structured data to be interpolated, and the greater the impact on the structured data to be interpolated, so the corresponding data weight is greater.
- the data weights corresponding to the structured data with known classification results are multiplied by the classification results to obtain the classification results corresponding to the structured data to be interpolated. Then, the structured data to be interpolated after the classification result is determined is inserted into the structured data of the known classification result, thereby solving the problem of missing data.
- the probability value of structured data A belonging to the real label category is 0.87
- the probability value of belonging to the predicted label category is 0.27
- the method includes: separately counting the first data volume corresponding to the noise data, and the structure The second data volume corresponding to the structured data, and according to the first data volume and the second data volume, calculate the first data ratio between the noise data and the structured data; if the first data If the ratio is greater than the preset noise data ratio, it is determined that the structured data does not meet the data pollution evaluation standard.
- the preset noise data ratio may be set according to actual service requirements.
- the preset noise data ratio is set to be 10%
- the total amount of statistical noise data (first data volume) is 200
- the total amount of structured data in the training data set ( The second data amount) is 1000
- the data ratio of 20% is greater than the preset noise data ratio of 10%, so it is determined that the training data set does not meet the data pollution evaluation criteria, that is, the training data set has quality problems and cannot be used for model training.
- the data to be evaluated is unstructured data in the prediction data set
- it is necessary to detect whether there is confrontation data in the data to be evaluated that is, to detect whether there is The data maliciously created by the attacker, because once such confrontation data is mixed into the prediction data set, it will directly affect the prediction result of the model.
- step 202 specifically includes: using The first preset compressor and the second preset compressor compress the unstructured data to obtain first compressed data and second compressed data corresponding to the unstructured data; data, the first compressed data, and the second compressed data to obtain the original prediction result corresponding to the unstructured data, the first prediction result corresponding to the first compressed data, and the second compressed data A second prediction result corresponding to the data; respectively calculating a first difference between the original prediction result and the first prediction result, and a second difference between the original prediction result and the second prediction result ; Based on the first difference and the second difference, determine whether the unstructured data is adversarial data.
- the first preset compressor and the second preset compressor can compress the features in the unstructured data to reduce the input of unnecessary features and reduce the dimension corresponding to the unstructured data.
- the first preset compressor The compressed features are different from those compressed by the second preset compressor. For example, if the input unstructured data includes 10 features, that is, the input dimension corresponding to the unstructured data is 10, the first preset compressor can The first feature and the second feature in the unstructured data are compressed, and the second preset compressor can compress the third feature and the fourth feature in the unstructured data.
- the number of compressors used in the embodiment of the present invention is not limited to two, and the number of compressors can be set according to actual service requirements and the number of features.
- the original prediction result, the first prediction result and the second prediction result in the embodiment of the present invention are the probability values that the unstructured data belongs to the corresponding label category.
- the unstructured data A is respectively input into the first preset compressor and the second preset compressor for feature compression processing, and the first compressed data and the second compressed data corresponding to the unstructured data A are obtained, and then the The unstructured data A, the first compressed data and the second compressed data were respectively input into the built model for prediction, and the original prediction result corresponding to the structured data A was 0.78, the first prediction result corresponding to the first compressed data was 0.56, and the first prediction result corresponding to the first compressed data was 0.56.
- the second predicted result corresponding to the second compressed data is 0.63. Further, the original predicted result is subtracted from the first predicted result to obtain the first difference of 0.22, and the original predicted result is subtracted from the second predicted result to obtain the second difference.
- the value is 0.15, and then a maximum difference is selected from the first difference and the second difference to compare with the preset difference. If the maximum difference is greater than the preset difference, then it is determined that the unstructured data A is confrontational data; If the maximum difference is less than the preset difference, it is determined that the unstructured data A is not confrontational data. For example, if the preset difference is set to 0.2, since the maximum difference 0.22 is greater than the preset difference 0.2, it is determined that the unstructured data is against data. All adversarial data present in the prediction data set can thus be determined in the manner described above.
- the method includes: separately counting the third data volume corresponding to the confrontation data, and a fourth data volume corresponding to the unstructured data, and calculate a second data ratio between the confrontation data and the unstructured data according to the third data volume and the fourth data volume; if If the second data ratio is greater than the preset confrontation data ratio, it is determined that the unstructured data does not meet the data pollution evaluation standard.
- the preset confrontation data ratio can be set according to actual business requirements.
- step 203 specifically includes: determining each feature corresponding to the structured data; using the preset bias corpus Preliminary detection of the bias features existing in the various features, and respectively excluding the bias features and their corresponding structured data from the various features and the training data set, to obtain the excluded features and the excluded structured data. Data; according to the excluded structured data, analyze whether each feature after the exclusion is a bias feature.
- the prediction bias corpus stores a large number of bias features, such as gender, age, region, income, etc.
- the features corresponding to the structured data in the training data set include: education level, work, income, medical history, and gender. Match the above features corresponding to the structured data with each feature in the preset bias corpus. Through matching, it can be found that, Among the features corresponding to the structured data, the income feature and the gender feature are bias features. In order to improve the detection accuracy of the bias feature, it is necessary to further analyze whether other features are bias features, and exclude the structured data corresponding to the bias feature from the structured data. Then use the excluded structured data to analyze whether the remaining features are bias features.
- the method includes: if there is a corresponding label category in the excluded structured data, then After each feature is combined with each label category, multiple combination results are obtained; the feature values corresponding to each feature after the exclusion are determined, and according to the multiple combination results, the corresponding feature values of each feature value under different label classifications are analyzed.
- First data volume distribution information based on the first data volume distribution information, determine whether each feature after the exclusion is a biased feature.
- the label categories of the excluded structured data include “yes” and “no”
- the excluded features include “education level” and "job”.
- the above features we get "education level- Yes”, “Education Level-No”, “Work-Yes” and “Work-No”, and then determine that the eigenvalues corresponding to the education level include undergraduates and above, undergraduates, and below undergraduates, and the eigenvalues corresponding to work include working and no work , further, first analyze the amount of data in the structured data whose label category is "yes" and the educational level is above undergraduate, undergraduate, and below.
- the data volume of statistical education is 1000 respectively. People, 200 people and 800 people, the total amount of structured data with the label category "Yes" is 2000 people.
- the proportions of unstructured data with a bachelor degree or above, bachelor degree and below are 50%, 10% and 40%, because the difference between the data volume of undergraduates and above is 40%, which exceeds the preset proportion of 20%, so it can be determined that the characteristic education level is a biased feature. In this way, it can be determined whether the remaining features corresponding to the structured data with label categories are biased features.
- the method further includes: if the excluded structured data does not have a corresponding label category, using the preset The clustering algorithm clusters the excluded structured data to obtain structured data under different classifications; determines the eigenvalues corresponding to each feature after the exclusion, and analyzes the corresponding eigenvalues of each eigenvalue under different classifications.
- Second data volume distribution information based on the second data volume distribution information, determine whether each feature after the exclusion is a bias feature.
- the preset clustering algorithm may specifically be a DBSCAN clustering algorithm.
- the excluded structured data may not have corresponding label categories.
- label categories and features cannot be combined to analyze the distribution of the first data volume corresponding to each feature value under different label categories.
- you can cluster the excluded structured data and then analyze the distribution of the second data volume corresponding to each feature value under different categories, and use the DBSCAN clustering algorithm to exclude
- the process of clustering the final structured data first set the neighborhood radius corresponding to the structured data and the threshold of the amount of structured data in the field, then select a structured data A arbitrarily, and calculate the arrival of each structured data
- the distance of the structured data determine the structured data B, C, and D included in the neighborhood corresponding to the structured data A, if the structured data included in the neighborhood corresponding to the structured data A If the quantity is greater than the structured data volume threshold, then determine structured data A as the core point, and build cluster C1 with structured data A as the core point, find out all the points reachable from structured data A density, structured
- the density of data A can reach the density of structured data B, and the density of structured data B can reach the density of structured data E, so the density of structured data A can reach the density of structured data E, that is, structured data E also belongs to C1, so according to the above method, it can Find all the structured data in cluster C1, and continue to search for other data in the excluded structured data.
- cluster C2 can be obtained, and the clustering of the excluded structured data can be completed by dividing into multiple clusters. Divide the excluded structured data into multiple categories, and then analyze the distribution of the second data volume corresponding to each feature value under different classifications. The method of analyzing the distribution of the second data volume is the same as that of the first data volume. This will not be repeated here.
- the attribute characteristics of the data to be evaluated under the data size evaluation dimension include the total amount, the number of features, the memory size occupied by the data, and whether the data has labels, etc.
- the attribute characteristics of the data to be evaluated under the data size evaluation dimension include the total amount, the number of features, the memory size occupied by the data, and whether the data has labels, etc.
- the amount of data corresponding to the data to be evaluated is 300
- the occupied memory size is 11.3KB
- the number of features is 13, and the data has labels.
- the scale evaluation of the data to be evaluated is required.
- the standards for data size evaluation are different. For example, for translation models, if the total amount of data corresponding to the data to be evaluated is less than 10 million, it is determined that the data to be evaluated does not meet the data size evaluation standard, that is, it does not To be able to directly use this data set for training or prediction, the number needs to be increased.
- the balance of the data set is an important factor affecting the effect of the artificial intelligence algorithm.
- the more uniform the data set the smaller the distribution deviation of the data to be evaluated, and the better the operating effect of the artificial intelligence algorithm.
- the distribution deviation of the data to be evaluated is The bigger it is, the worse the AI algorithm will perform.
- the label categories corresponding to the data to be evaluated include “yes” and “no”, respectively count the amount of data with the label category "yes” and the data volume with the label category “no”, and calculate the proportion of the number of different label categories , such as calculating the proportion of the data volume corresponding to the "Yes” label is 90%, the proportion of data volume corresponding to the "No” label is 10%, and the difference between the data volume proportions of the two types of labels reaches 80%.
- the difference in the ratio is greater than 60% of the difference in the ratio of the preset data volume, so it is determined that the data to be evaluated does not meet the data balance evaluation standard.
- the attribute characteristics of the data to be evaluated under the data accuracy evaluation dimension include the total amount of data, the total amount of missing tags, and whether various tags are abnormal. For example, the total number of missing labels in the data to be evaluated is counted, and then the ratio of the total amount of data with missing labels to the total amount of data to be evaluated is calculated. If the ratio is greater than the preset ratio, it is determined that the data to be evaluated does not meet the accuracy requirements. gender assessment criteria. Another example is to count the data volumes corresponding to the labels "Yes" and "No". If the data volume under a certain label is less than the preset data volume, it is determined that the label is abnormal, and then it can be determined that there are abnormal labels in the data to be evaluated. Accuracy Evaluation Criteria.
- the quality evaluation results of the data to be evaluated in multiple evaluation dimensions are obtained, and then a quality evaluation report corresponding to the data to be evaluated is generated for reference by technical personnel. It should be noted that the execution order of the above steps 202-206 is not limited to the order shown in FIG. It can be executed in parallel, which is not limited by the present invention.
- Another data set quality assessment method provided by the embodiment of the present invention compared with the current method of assessing the quality of the data set by technicians based on their own experience, this method can obtain the data to be evaluated in the data set; and Statistics of the attribute characteristics of the data to be evaluated under multiple evaluation dimensions; at the same time, based on the attribute characteristics of the multiple evaluation dimensions, the quality of the data to be evaluated is evaluated, and the data to be evaluated are respectively obtained.
- the quality evaluation results under the multiple evaluation dimensions so by counting the attribute characteristics of the data to be evaluated under the multiple evaluation dimensions, the quality of the data set can be automatically evaluated from multiple evaluation dimensions, thereby improving the quality of the data set
- the evaluation accuracy and evaluation efficiency effectively ensure the safety of data sets in the process of artificial intelligence development.
- an embodiment of the present invention provides a data set quality assessment device. As shown in FIG. 3 , the device includes: an acquisition unit 31 , a statistical unit 32 and an evaluation unit 33 .
- the obtaining unit 31 may be used to obtain the data to be evaluated in the data set.
- the statistics unit 32 may be used to separately calculate attribute characteristics of the data to be evaluated under multiple evaluation dimensions.
- the evaluation unit 33 may be configured to perform quality evaluation on the data to be evaluated based on attribute characteristics under the multiple evaluation dimensions, and obtain quality evaluation results of the data to be evaluated under the multiple evaluation dimensions respectively .
- the statistical unit 32 can specifically be used to separately count the data to be evaluated in the data scale evaluation dimension, data balance evaluation dimension, data accuracy evaluation dimension, data pollution evaluation dimension, and data bias evaluation dimension properties below.
- the evaluation unit 33 can be specifically configured to evaluate attributes based on the data scale evaluation dimension, the data balance evaluation dimension, the data accuracy evaluation dimension, the data pollution evaluation dimension, and the data bias evaluation dimension features, performing quality assessment on the data to be evaluated, and obtaining the data to be evaluated in the data scale evaluation dimension, the data balance evaluation dimension, the data accuracy evaluation dimension, the data pollution evaluation dimension and Quality assessment results under the data bias assessment dimension
- the statistical unit 32 includes: a fitting module 321 , a prediction module 322 and a judgment module 323 .
- the fitting module 321 can be configured to use a preset interpolation algorithm to fit a function curve corresponding to the structured data according to the structured data and its corresponding tag category.
- the prediction module 322 may be configured to use the function curve to predict the structured data to obtain the predicted label category corresponding to the structured data.
- the determination module 323 may be configured to determine whether the structured data is noise data based on the predicted label category and the label category corresponding to the structured data.
- the evaluation unit 33 includes: a first calculation module 331 and a first determination module 332 .
- the first calculation module 331 may be configured to separately count the first data volume corresponding to the noise data and the second data volume corresponding to the structured data, and calculate according to the first data volume and the second data volume , calculating a first data ratio between the noise data and the structured data.
- the first determining module 332 may be configured to determine that the structured data does not satisfy the data pollution evaluation standard when the first data ratio is greater than a preset noise data ratio.
- the statistical unit 32 further includes: a compression module 324 and a second calculation module 325 to make statistics on the attribute characteristics of the data to be evaluated under the data pollution evaluation dimension.
- the compression module 324 may be configured to compress the unstructured data by using the first preset compressor and the second preset compressor respectively, to obtain the first compressed data and the second compressed data corresponding to the unstructured data. Two compressed data.
- the prediction module 322 may be configured to respectively predict the unstructured data, the first compressed data, and the second compressed data to obtain an original prediction result corresponding to the unstructured data, and the first A first prediction result corresponding to the compressed data, and a second prediction result corresponding to the second compressed data.
- the second calculation module 325 may be configured to calculate a first difference between the original prediction result and the first prediction result, and a first difference between the original prediction result and the second prediction result, respectively. Two difference.
- the determining module 323 may be configured to determine whether the unstructured data is confrontational data based on the first difference and the second difference.
- the first calculation module 331 can also be used to separately count the third data volume corresponding to the confrontation data and the fourth data volume corresponding to the unstructured data, and according to the third data volume and the The fourth data volume is to calculate a second data ratio between the confrontation data and the unstructured data.
- the first determining module 332 may also be configured to determine that the unstructured data does not meet the data pollution evaluation standard if the second data ratio is greater than a preset confrontation data ratio.
- the statistics unit 32 in order to count the attribute characteristics of the data to be evaluated under the data bias evaluation dimension, the statistics unit 32 further includes: a second determination module 326 , an exclusion module 327 and an analysis module 328 .
- the second determination module 326 may be used to determine each feature corresponding to the structured data.
- the exclusion module 327 can be used to preliminarily detect the biased features in the various features by using the preset biased corpus, and exclude the biased features and their corresponding structured features from the various features and the training data set. data to obtain the excluded features and the excluded structured data.
- the analysis module 328 can be configured to analyze whether each feature after exclusion is a bias feature according to the structured data after exclusion.
- the analysis module 328 can be used to, if there is a corresponding label category in the structured data after exclusion, then Each feature of each feature is combined with each label category to obtain multiple combination results; determine the feature value corresponding to each feature after the exclusion, and according to the multiple combination results, analyze the No. 1 corresponding to each feature value under different label classifications A data volume distribution information; based on the first data volume distribution information, it is determined whether each feature after the exclusion is a biased feature.
- the analysis module 328 can also be used to use the preset aggregation
- the class algorithm clusters the excluded structured data to obtain structured data under different classifications; determines the eigenvalues corresponding to each feature after the exclusion, and analyzes the first eigenvalues corresponding to each eigenvalue under different classifications.
- Data volume distribution information Based on the second data volume distribution information, determine whether each feature after the exclusion is a biased feature.
- an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: obtaining the waiting Evaluate the data; respectively count the attribute characteristics of the data to be evaluated under multiple evaluation dimensions; perform quality assessment on the data to be evaluated based on the attribute characteristics under the multiple evaluation dimensions, and obtain the data to be evaluated respectively in Quality evaluation results under the multiple evaluation dimensions.
- the embodiment of the present invention also provides a physical structure diagram of a computer device, as shown in FIG. 5 , the computer device includes: a processor 41, A memory 42 and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on the bus 43 .
- the processor 41 executes the program, the following steps are implemented: respectively counting the attribute characteristics of the data to be evaluated under multiple evaluation dimensions; based on the attribute characteristics under the multiple evaluation dimensions, performing The quality assessment is to obtain the quality assessment results of the data to be assessed under the plurality of assessment dimensions respectively.
- Fangming can obtain the data to be evaluated in the data set; and separately count the attribute characteristics of the data to be evaluated in multiple evaluation dimensions; at the same time, based on the data in the multiple evaluation dimensions attribute characteristics, performing quality assessment on the data to be evaluated, and obtaining quality evaluation results of the data to be evaluated in the plurality of evaluation dimensions, and thus by counting the attribute characteristics of the data to be evaluated in the plurality of evaluation dimensions,
- the quality of the data set can be automatically evaluated from multiple evaluation dimensions, thereby improving the evaluation accuracy and efficiency of the data set quality, and effectively ensuring the safety of the data set in the process of artificial intelligence development.
- each module or each step of the above-mentioned present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network formed by multiple computing devices Alternatively, they may be implemented in program code executable by a computing device so that they may be stored in a storage device to be executed by a computing device, and in some cases in an order different from that shown here
- the steps shown or described are carried out, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation.
- the present invention is not limited to any specific combination of hardware and software.
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种数据集质量评估方法、装置、计算机设备及存储介质,涉及信息技术领域,主要在于能够提高数据集质量的评估精度和评估效率。其中方法包括:获取数据集中的待评估数据;分别统计所述待评估数据在多个评估维度下的属性特征;基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。本发明适用于数据集质量的评估。
Description
本发明涉及信息技术领域,尤其是涉及一种数据集质量评估方法、装置、计算机设备及存储介质。
数据是人工智能开发和应用的基础,数据集对于人工智能算法来说至关重要,使用不同质量的数据集会在训练之后得到不同的模型参数,产生不同的执行效果,进而影响人工智能算法的安全性,如果非法分子利用攻击手段对数据集进行恶意修改和添加,会导致模型预测出错,因此如何有效地对数据集质量进行检测和评估,成为了人工智能安全亟需解决的问题。
目前,通常由技术人员根据各自的经验对数据集的质量进行评估。然而,这种质量评估方式,较为依赖技术人员的工作经验,评估结果受人为主观因素的影响较大,因此很可能无法对数据集的质量作出准确评估,进而造成人工智能安全事故,此外,这种人为评估数据集质量的方式,评估效率较低,而且增加了技术人员的工作量。
发明内容
本发明提供了一种数据集质量评估方法、装置、计算机设备及存储介质,主要在于能够提高数据集质量的评估精度和评估效率。
根据本发明的第一个方面,提供一种数据集质量评估方法,包括:
获取数据集中的待评估数据;
分别统计所述待评估数据在多个评估维度下的属性特征;
基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
根据本发明的第二个方面,提供一种数据集质量评估装置,包括:
获取单元,用于获取数据集中的待评估数据;
统计单元,用于分别统计所述待评估数据在多个评估维度下的属性特征;
评估单元,用于基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
根据本发明的第三个方面,提供一种计算机可读存储介质,其上存储有计算机程序,该程序被处理器执行时实现以下步骤:
获取数据集中的待评估数据;
分别统计所述待评估数据在多个评估维度下的属性特征;
基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估, 得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
根据本发明的第四个方面,提供一种计算机设备,包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,所述处理器执行所述程序时实现以下步骤:
获取数据集中的待评估数据;
分别统计所述待评估数据在多个评估维度下的属性特征;
基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
本发明提供的一种数据集质量评估方法、装置、计算机设备及存储介质,与目前由技术人员根据各自的经验对数据集的质量进行评估的方式相比,本方明能够获取数据集中的待评估数据;并分别统计所述待评估数据在多个评估维度下的属性特征;与此同时,基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果,由此通过统计待评估数据在多个评估维度下的属性特征,能够从多个评估维度对数据集的质量进行自动评估,从而能够提高数据集质量的评估精度和评估效率,有效地保证了人工智能开发过程中数据集的安全。
此处所说明的附图用来提供对本发明的进一步理解,构成本申请的一部分,本发明的示意性实施例及其说明用于解释本发明,并不构成对本发明的不当限定。在附图中:
图1示出了本发明实施例提供的一种数据集质量评估方法流程图;
图2示出了本发明实施例提供的另一种数据集质量评估方法流程图;
图3示出了本发明实施例提供的一种数据集质量评估装置的结构示意图;
图4示出了本发明实施例提供的另一种数据集质量评估装置的结构示意图;
图5示出了本发明实施例提供的一种计算机设备的实体结构示意图。
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是,在不冲突的情况下,本申请中的实施例及实施例中的特征可以相互组合。
目前,通常由技术人员根据各自的经验对数据集的质量进行评估。然而,这种质量评估方式,较为依赖技术人员的工作经验,评估结果受人为主观因素的影响较大,因此很可能无法对数据集的质量作出准确评估,进而造成人工智能安全事故,此外,这种人为评估数据集质量的方式,评估效率较低,而且增加了技术人员的工作量。
为了解决上述问题,本发明实施例提供了一种数据集质量评估方法,如图1所示,所述方法包括:
101、获取数据集中的待评估数据。
其中,数据集包括训练数据集和预测数据集,待评估数据具体可以为训练数据集中的各个样本数据,或者预测数据集中的各个预测数据。为了克服现有技术中数据集质量的评估精度和评估效率较低的缺陷,本发明实施例开发了一套数据集质量评估工具,通过统计待评估数据在多个评估维度下的属性特征,能够从多个评估维度对数据集的质量进行自动评估,进而提高了数据集质量的评估精度和评估效率,同时保证了人工智能开发过程中数据集的安全。本发明实施例主要应用于对数据集质量进行多维度评估的场景。本发明实施例的执行主体为能够对数据集质量进行评估的装置或者设备,具体可以设置在服务器一侧。
对于本发明实施例,预先收集需要进行质量评估的训练数据集和预测数据集,该训练数据集和预测数据集中的数据具体可以为结构化数据,也可以为非结构化数据,如图像数据。在获取待评估的训练数据集或者预测数据集之后,技术人员可以点击数据集质量评估工具界面的文件上传按钮,将待评估的训练数据集或者预测数据集上传至数据集质量评估工具中,以便数据集质量评估工具对待评估的数据集进行多维度的质量评估。
102、分别统计所述待评估数据在多个评估维度下的属性特征。
其中,多个评估维度包括数据规模评估维度、数据均衡性评估维度、数据准确性评估维度、数据污染评估维度和数据偏见评估维度,需要说明的是,本发明实施例中的评估维度并不局限于上述列举出来的评估维度,还可以包括其他评估维度,具体可以根据实际的业务需求进行设定。进一步地,待评估数据在数据规模评估维度下的属性特征包括:数据总量、特征数量、数据所占内存大小、数据是否有标签等;待评估数据在数据均衡下评估维度下的属性特征包括各类标签下的数据量占比;待评估数据在数据准确性评估维度下的属性特征包括:数据总量、标签缺失总量、各类标签是否异常等;待评估数据在数据污染评估维度下的属性特征包括:噪声数据量、对抗数据量等;待评估数据在数据偏见评估维度下的属性特征包括待评估数据对应的偏见特征。
对于本发明实施例,可以采用不同的统计方式,分别统计待评估数据在数据规模评估维度、数据均衡性评估维度、数据准确性评估维度、数据污染评估维度和数据偏见评估维度下的属性特征,针对不同评估维度所采用的具体统计方式不同,具体见步骤202-206。
103、基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
对于本发明实施例,不同评估维度对应的评估标准不同,在利用多个评估维度下的属性特征对待评估数据进行质量评估的过程中,如果待评估数据不满足任意一个评估维度对应的评估标准,则确定待评估数据存在质量问题,无法利用该数据集对模型进行训练或者预测,技术人员需要重新收集数据集或者对存在质量问题的数据集进行数据清洗。例如,待评估数据在数据均衡性评估维度下的“是”标签对应的数据量占比为90%,“否”标签对应的数据量占比为10%,两类标签的数据量占比之差达到80%,由于该数据量占比之差大于预设数据量占比之差60%,因此确定待评估数据不满足数据均衡性评估标准,如果利用这种不满足数据均衡性评估标准的数据对模型进行训练,很可能影响模型的执行效果,无法保证人工智能算法的安全,同理还可以从数据规模评估维度、数据准确性评估维度、数据污染评估维度、数据偏见评估维度对待评估数据进行质量评估,若待评估数据不满足上述维度对应的评估标准,则确定待评估数据存在质量问题,无法利用其对模型进行训练或者预测,针对不同评估维度的质量评估过程具体见步骤202-206。
本发明实施例提供的一种数据集质量评估方法,与目前由技术人员根据各自的经验对数据集的质量进行评估的方式相比,本方明能够获取数据集中的待评估数据;并分别统计所述待评估数据在多个评估维度下的属性特征;与此同时,基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果,由此通过统计待评估数据在多个评估维度下的属性特征,能够从多个评估维度对数据集的质量进行自动评估,从而能够提高数据集质量的评估精度和评估效率,有效地保证了人工智能开发过程中数据集的安全。
进一步的,为了更好的说明上述数据集的质量评估过程,作为对上述实施例的细化和扩展,本发明实施例提供了另一种污染样本数据的检测方法,如图2所示,所述方法包括:
201、获取数据集中的待评估数据。
其中,数据集包括训练数据集和预测数据集,待评估数据具体可以为训练数据集中的各个样本数据,或者预测数据集中的各个预测数据。对于本发明实施例,在对数据集进行质量评估之前,需要获取待评估的数据集,数据集的具体获取方式与步骤101完全相同,在此不再赘述。
202、统计所述待评估数据在数据污染评估维度下的属性特征,并基于所述数据污染评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据污染评估维度下的质量评估结果。
对于本发明实施例,如果待评估数据是训练数据集中的结构化数据,则在对待评估数据进行污染评估的过程中,需要检测待评估数据中是否存在噪声数据,因为噪声数据对模型训练的干扰较大,容易影响模型的执行效果,针对识别噪声数据的具体过程,作为一种可选实施方式,步骤202具体包括:根据所述结构化数据及其对应的标签类别,利用预设插值算法拟合所述结构化数据对应的函数曲线;利用所述函数曲线对所述结构化数据进行预测,得到所述结构化数据对应的预测标签类别;基于所述预测标签类别和所述结构化数据对应的标签类别,判定所述结构化数据是否为噪声数据。
具体地,训练数据集中存在大量的结构化数据,将每一个结构化数据作为一个样本点(x,y),并利用这些样本点(x
1,y
1),(x
2,y
2),…(x
n,y
n),来拟合一个函数曲线y=f(x),由于待评估的结构化数据可能存在数据缺陷的现象,因此在利用大量样本点拟合函数曲线之前,需要利于预设插值算法对结构化数据进行插值处理,该预设插值算法具体可以为预设克里金插值算法,首先,确定待插值的结构化数据,并计算待插值的结构化数据与这些已知分类结果的结构化数据之间的距离,并根据该距离确定已知分类结构的结构化数据对应的数据权重,之后根据已知分类结果的结构化数据对应的权重和该分类结果,计算待插值的结构化数据对应的分类结果。其中,该分类结果具体可以为结构化数据对应的分类概率值。
例如,已知分类结果的结构化数据为x1,x2,x3,确定待插值的结构化数据为x4,由于待插值的结构数据x4对应的分类结果是未知的,因此可以利用已知分类结果的结构化数据x1,x2,x3,预估待插值的结构化数据x4对应的分类结果,具体地,分别计算待插值的结构化数据x4与已知分类结果的结构化数据x1,x2,x3之间的距离,距离越大,说明该已知分类结果的结构化数据与待插值的结构化数据相距越远,其对待插值的结构化数据的影响较小,因此其对应的数据权重越小;相反距离越小,说明该已知分类结果的结构化数据与待插值的结构化数据相距越近,其对待插值的结构化数据的影响较大,因此其对应的数据权重越大,进一步地,在确定已知分类结果的结构化数据对应的数据权重之后,将已知分类结果的结构化数据对应的数据权重和分类结果相乘,得到待插值的结构化数据对应的分类结果。接着将确定完分类结果的待插值的结构化数据***至已知分类结果的结构化数据中,由此能够解决数据缺失的问题。
进一步地,将上述结构化数据共同作为样本点,利用这些样本点进行曲线拟合,得到函数曲线y=f(x),由于训练数据集中结构化数据对应的分类结果 是已知的,即属于真实标签类别的概率值是已知的,接下来可以利用函数曲线对上述的结构化数据进行预测,得到结构化数据对应的预测结果,即属于预测标签类别的概率值。进一步地,将结构化数据属于预测标签类别的概率值与属于真实标签类别的概率值相减,得到结构化数据对应的概率差,如果该概率差大于预设概率差,则确定该结构化数据属于噪声数据。例如,结构化数据A属于真实标签类别的概率值为0.87,属于预测标签类别的概率值为0.27,概率差为0.87-0.27=0.5,由于该概率差大于预设概率差0.2,因此确定结构化数据A为噪声数据。由此按照上述方式能够确定训练数据集中的每个结构化数据是否为噪声数据。
进一步地,在确定训练数据集中的噪声数据之后,需要对该训练数据集中的结构化数据进行污染评估,基于此,所述方法包括:分别统计所述噪声数据对应的第一数据量,以及结构化数据对应的第二数据量,并根据所述第一数据量和所述第二数据量,计算所述噪声数据与所述结构化数据之间的第一数据比例;若所述第一数据比例大于预设噪声数据比例,则确定所述结构化数据不满足数据污染评估标准。其中,该预设噪声数据比例可以根据实际的业务需求进行设定。
例如,设定预设噪声数据比例为10%,在确定训练数据集中存在的噪声数据之后,统计噪声数据的总量(第一数据量)为200个,训练数据集中结构化数据的总量(第二数据量)为1000个,由此能够计算噪声数据对应的第一数据量与结构化数据对应的第二数据量之间的第一数据比例为200/1000=20%,由于该第一数据比例20%大于预设噪声数据比例10%,因此确定该训练数据集不满足数据污染评估标准,即该训练数据集存在质量问题,不能够用于模型训练。
在具体应用场景中,如果待评估数据是预测数据集中的非结构化数据,则对待评估数据进行污染评估的过程中,需要检测待评估数据中是否存在对抗数据,即检测待评估数据中是否存在攻击者恶意打造的数据,因为预测数据集中一旦混入这种对抗数据,将会直接影响模型的预测结果,针对识别对抗数据的具体过程,作为一种可选实施方式,步骤202具体包括:分别利用第一预设压缩器和第二预设压缩器对所述非结构化数据进行压缩处理,得到所述非结构化数据对应的第一压缩数据和第二压缩数据;分别对所述非结构化数据、所述第一压缩数据和所述第二压缩数据进行预测,得到所述非结构化数据对应的原始预测结果,所述第一压缩数据对应的第一预测结果,以及所述第二压缩数据对应的第二预测结果;分别计算所述原始预测结果与所述第一预测结果之间的第一差值,以及所述原始预测结果与所述第二预测结果之间的第二差值;基于所述第一差值和所述第二差值,判定所述非结构化数据是否为对抗数据。
其中,第一预设压缩器和第二预设压缩器能够对非结构化数据中的特征进行压缩,以减少不必要特征的输入,降低非结构化数据对应的维度,第一预设压缩器所压缩的特征与第二预设压缩器所压缩的特征并不相同,如输入的非结构化数据包括10个特征,即非结构化数据对应的输入维度是10,第一预设压缩器能够对非结构化数据中的第一个特征和第二个特征进行压缩,第二预设压缩器能够对非结构化数据中的第三个特征和第四个特征进行压缩。需要说明的是,本发明实施例中所采用的压缩器的数量并不局限于两个,具体可以根据实际的业务需求和特征数量,设定压缩器的数量。此外,本发明实施例中的原始预测结果、第一预测结果和第二预测结果为非结构化数据属于相应标签类别的概率值。
例如,将非结构化数据A分别输入至第一预设压缩器和第二预设压缩器中进行特征压缩处理,得到非结构化数据A对应的第一压缩数据和第二压缩数据,之后将非结构化数据A、第一压缩数据和第二压缩数据分别输入至构建的模型中进行预测,得到结构化数据A对应的原始预测结果0.78,第一压缩数据对应的第一预测结果0.56,第二压缩数据对应的第二预测结果0.63,进一步地,将原始预测结果与第一预测结果相减,得到第一差值为0.22,将原始预测结果与第二预测结果相减,得到第二差值为0.15,之后从第一差值和第二差值中选择一个最大差值与预设差值进行比较,如果最大差值大于预设差值,则确定非结构化数据A为对抗数据;如果最大差值小于预设差值,则确定非结构化数据A不是对抗数据,如设定预设差值为0.2,由于最大差值0.22大于预设差值0.2,因此确定非结构化数据是对抗数据。由此按照上述方式能够确定预测数据集中存在的所有对抗数据。
进一步地,在确定预测数据集中的对抗数据之后,需要对该预测数据集中的非结构化数据进行污染评估,基于此,所述方法包括:分别统计所述对抗数据对应的第三数据量,以及所述非结构化数据对应的第四数据量,并根据所述第三数据量和所述第四数据量,计算所述对抗数据与所述非结构化数据之间的第二数据比例;若所述第二数据比例大于预设对抗数据比例,则确定所述非结构化数据不满足数据污染评估标准。其中,该预设对抗数据比例可以根据实际的业务需求进行设定。
例如,设定预设对抗数据比例为10%,在确定预测数据集中存在的对抗数据之后,统计对抗数据的总量(第三数据量)为300个,预测数据集中非结构化数据的总量(第四数据量)为1000个,由此能够计算对抗数据对应的第三数据量与非结构化数据对应的第四数据量之间的第二数据比例为300/1000=30%,由于该第二数据比例30%大于预设对抗数据比例10%,因此确定该预测数据集不满足数据污染评估标准,即该预测数据集存在质量问题,不能够用于模型预测。
203、统计所述待评估数据在数据偏见评估维度下的属性特征,并基于所述数据偏见评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据偏见评估维度下的质量评估结果。
对于本发明实施例,如果待评估数据是训练数据集中的结构化数据,则在对待评估数据进行偏见评估的过程中,需要检测待评估数据对应的各个特征是否为偏见特征,因为偏见特征的存在,可能会导致人工智能的决策结果带有歧视,针对确定偏见特征的具体过程,作为一种可选实施方式,步骤203具体包括:确定所述结构化数据对应的各个特征;利用预设偏见语料库初步检测所述各个特征中存在的偏见特征,并从所述各个特征和所述训练数据集中分别排除所述偏见特征及其对应的结构化数据,得到排除后的各个特征和排除后的结构化数据;根据所述排除后的结构化数据,分析所述排除后的各个特征是否为偏见特征。其中,预测偏见语料库中存储有大量偏见特征,如性别、年龄、地域、收入等。
例如,训练数据集中的结构化数据对应的特征包括:教育程度、工作、收入、病史、性别,将结构化数据对应的上述特征与预设偏见语料库中的各个特征进行匹配,通过匹配可以发现,结构化数据对应的特征中收入特征和性别特征为偏见特征,为提高偏见特征的检测精度,还需要进一步分析其他特征是否为偏见特征,从结构化数据中排除该偏见特征对应的结构化数据,之后利用排除后的结构化数据,分析剩余的各个特征是否为偏见特征。
进一步地,针对分析剩余的各个特征是否为偏见特征的具体过程,作为一种可选实施方式,所述方法包括:若所述排除后的结构化数据存在相应的标签类别,则将所述排除后的各个特征与各个标签类别进行组合,得到多个组合结果;确定所述排除后的各个特征对应的特征值,并根据所述多个组合结果,分析在不同标签分类下各个特征值对应的第一数据量分布信息;基于所述第一数据量分布信息,判定所述排除后的各个特征是否为偏见特征。
例如,排除后的结构化数据存在的标签类别包括“是”和“否”,排除后的各个特征包括“教育程度”和“工作”,将上述特征与标签类别进行组合,得到“教育程度-是”、“教育程度-否”、“工作-是”和“工作-否”,之后确定教育程度对应的特征值包括本科以上、本科、本科以下,工作对应的特征值包括有工作和无工作,进一步地,首先分析标签类别为“是”的结构化数据中教育程度分别为本科以上、本科、本科以下的数据量,如统计教育程度为本科以上、本科、本科以下的数据量分别为1000人,200人和800人,标签类别为“是”的结构化数据总量为2000人,由此可知,本科以上、本科、本科以下非结构化数据的占比分别为50%,10%和40%,由于本科以上的数据量与本科的数据量占比之差为40%超过预设占比之差20%,因此可以确定特征教育程度为偏见特征。由此按照上述方式能够确定存在标签类别的结构化数据对应的剩余特征 是否为偏见特征。
进一步地,针对分析剩余的各个特征是否为偏见特征的具体过程,作为一种可选实施方式,所述方法还包括:若所述排除后的结构化数据不存在相应的标签类别,利用预设聚类算法对所述排除后的结构化数据进行聚类处理,得到不同分类下的结构化数据;确定所述排除后的各个特征对应的特征值,并分析在不同分类下各个特征值对应的第二数据量分布信息;基于所述第二数据量分布信息,判定所述排除后的各个特征是否为偏见特征。其中,该预设聚类算法具体可以为DBSCAN聚类算法。
具体地,排除后的结构化数据可能不存在相应的标签类别,此时便无法将标签类别和特征进行组合,分析不同标签类别下各个特征值对应的第一数据量分布,因此在不知道结构化数据对应的标签类别的情况下,可以通过对排除后的结构化数据进行聚类,之后再分析在不同分类下各个特征值对应的第二数据量分布情况,在利用DBSCAN聚类算法对排除后的结构化数据进行聚类的过程中,首先设定结构化数据对应的邻域半径和领域中结构化数据量阈值,之后任意选择一个结构化数据A,并计算出每一个结构化数据到达该结构化数据的距离,根据计算的各个距离,确定结构化数据A对应的邻域中包括的各个结构化数据B、C、D,如果结构化数据A对应的邻域中包括的结构化数据数量大于结构化数据量阈值,则确定结构化数据A为核心点,并以结构化数据A为核心点建立簇C1,找出所有从结构化数据A密度可达的点,结构化数据A邻域内的所有结构化数据都是结构化数据A密度可达的点,都属于C1,此外,确定结构化数据B对应的邻域中包括的结构化数据,如包括结构化数据E,由于结构化数据A密度可达结构化数据B,结构化数据B密度可达结构化数据E,所以结构化数据A密度可达结构化数据E,即结构化数据E也属于C1,由此按照上述方式能够找到簇C1中的所有的结构化数据,继续在寻找排除后的结构化数据中的其他数据,按照上述方式能够得到簇C2,通过分成多个簇完成对排除后的结构化数据的聚类,即将排除后的结构化数据分成多个类别,之后再分析在不同分类下各个特征值对应的第二数据量分布情况,分析第二数据量分布情况的方式与第一数据量分布情况相同,在此不再赘述。
204、统计所述待评估数据在数据规模评估维度下的属性特征,并基于所述数据规模评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据规模评估维度下的质量评估结果。
其中,待评估数据在数据规模评估维度下的属性特征包括数量总量、特征数量、数据所占内存大小、数据是否有标签等。对于本发明实施例,在对待评估数据进行规模评估的过程中,需要统计待评估数据对应的数据总量,特征数量、所占内存大小,以及数据是否有标签,如果存在标签缺失的情况, 缺失标签的数据量为多少。例如,待评估数据对应的数据量为300,所占内存大小为11.3KB,特征数为13个,数据均有标签。
进一步地,在统计完成待评估数据对应的数量总量、特征数量、所占内存大小,以及是否有标签后,需要对待评估数据进行规模评估。需要说明的是,针对不同模型算法,数据规模评估的标准不同,例如,针对翻译模型,如果待评估数据对应的数据总量小于1000万,则确定待评估数据不满足数据规模评估标准,即不能够直接使用该数据集进行训练或者预测,需要增加数量。
205、统计所述待评估数据在数据均衡性评估维度下的属性特征,并基于所述数据均衡性评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据均衡性评估维度下的质量评估结果。
对于本发明实施例,数据集的均衡性是影响人工智能算法效果的一个重要因素,数据集越均匀,待评估数据分布偏差越小,人工智能算法的运行效果越好,相反待评估数据分布偏差越大,人工智能算法的运行效果越不好。例如,待评估数据对应的标签类别包括“是”和“否”,分别统计标签类别为“是”的数据量,以及标签类别为“否”的数据量,并计算不同标签类别的数量占比,如计算“是”标签对应的数据量占比为90%,“否”标签对应的数据量占比为10%,两类标签的数据量占比之差达到80%,由于该数据量占比之差大于预设数据量占比之差60%,因此确定待评估数据不满足数据均衡性评估标准。
206、统计所述待评估数据在数据准确性评估维度下的属性特征,并基于所述数据准确性评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据准确性评估维度下的质量评估结果。
其中,待评估数据在数据准确性评估维度下的属性特征包括数据总量、标签缺失总量和各类标签是否异常等。例如,统计待评估数据中缺失标签的数量总量,之后计算缺失标签的数据总量与待评估数据对应的数量总量的比值,如果该比值大于预设比值,则确定待评估数据不满足准确性评估标准。再比如,分别统计标签“是”和“否”对应的数据量,如果某一标签下的数据量小于预设数据量,则确定该标签异常,进而能够确定待评估数据存在异常标签,不满足准确性评估标准。
在对待评估数据进行多维度评估后,得到待评估数据分别在多个评估维度下的质量评估结果,进而生成待评估数据对应的质量评估报告,供技术人员参考。需要说明的是,以上202-206各个步骤的执行顺序并不以图2及前述的顺序为限,在具体应用中,202-206各个步骤可以根据实际情况按照其他顺序执行,当然202-206还可以并行执行,本发明对此不做限制。
本发明实施例提供的另一种数据集质量评估方法,与目前由技术人员根据各自的经验对数据集的质量进行评估的方式相比,本方明能够获取数据集中的待评估数据;并分别统计所述待评估数据在多个评估维度下的属性特征;与此同时,基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果,由此通过统计待评估数据在多个评估维度下的属性特征,能够从多个评估维度对数据集的质量进行自动评估,从而能够提高数据集质量的评估精度和评估效率,有效地保证了人工智能开发过程中数据集的安全。
进一步地,作为图1的具体实现,本发明实施例提供了一种数据集质量评估装置,如图3所示,所述装置包括:获取单元31、统计单元32和评估单元33。
所述获取单元31,可以用于获取数据集中的待评估数据。
所述统计单元32,可以用于分别统计所述待评估数据在多个评估维度下的属性特征。
所述评估单元33,可以用于基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
在具体应用场景中,所述统计单元32,具体可以用于分别统计所述待评估数据在数据规模评估维度、数据均衡性评估维度、数据准确性评估维度、数据污染评估维度、数据偏见评估维度下的属性特征。
所述评估单元33,具体可以用于基于所述数据规模评估维度、所述数据均衡性评估维度、所述数据准确性评估维度、所述数据污染评估维度和所述数据偏见评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述数据规模评估维度、所述数据均衡性评估维度、所述数据准确性评估维度、所述数据污染评估维度和所述数据偏见评估维度下的质量评估结果
进一步地,统计所述待评估数据在数据污染评估维度下的属性特征,如图4所示,所述统计单元32,包括:拟合模块321、预测模块322和判定模块323。
所述拟合模块321,可以用于根据所述结构化数据及其对应的标签类别,利用预设插值算法拟合所述结构化数据对应的函数曲线。
所述预测模块322,可以用于利用所述函数曲线对所述结构化数据进行预测,得到所述结构化数据对应的预测标签类别。
所述判定模块323,可以用于基于所述预测标签类别和所述结构化数据对应的标签类别,判定所述结构化数据是否为噪声数据。
基于此,所述评估单元33,包括:第一计算模块331和第一确定模块332。
所述第一计算模块331,可以用于分别统计所述噪声数据对应的第一数据量,以及结构化数据对应的第二数据量,并根据所述第一数据量和所述第二数据量,计算所述噪声数据与所述结构化数据之间的第一数据比例。
所述第一确定模块332,可以用于当所述第一数据比例大于预设噪声数据比例时确定所述结构化数据不满足数据污染评估标准。
进一步地,统计所述待评估数据在数据污染评估维度下的属性特征,所述统计单元32,还包括:压缩模块324和第二计算模块325。
所述压缩模块324,可以用于分别利用第一预设压缩器和第二预设压缩器对所述非结构化数据进行压缩处理,得到所述非结构化数据对应的第一压缩数据和第二压缩数据。
所述预测模块322,可以用于分别对所述非结构化数据、所述第一压缩数据和所述第二压缩数据进行预测,得到所述非结构化数据对应的原始预测结果,所述第一压缩数据对应的第一预测结果,以及所述第二压缩数据对应的第二预测结果。
所述第二计算模块325,可以用于分别计算所述原始预测结果与所述第一预测结果之间的第一差值,以及所述原始预测结果与所述第二预测结果之间的第二差值。
所述判定模块323,可以用于基于所述第一差值和所述第二差值,判定所述非结构化数据是否为对抗数据。
所述第一计算模块331,还可以用于分别统计所述对抗数据对应的第三数据量,以及所述非结构化数据对应的第四数据量,并根据所述第三数据量和所述第四数据量,计算所述对抗数据与所述非结构化数据之间的第二数据比例。
所述第一确定模块332,还可以用于若所述第二数据比例大于预设对抗数据比例,则确定所述非结构化数据不满足数据污染评估标准。
在具体应用场景中,为了统计所述待评估数据在数据偏见评估维度下的属性特征,所述统计单元32,还包括:第二确定模块326、排除模块327和分析模块328。
所述第二确定模块326,可以用于确定所述结构化数据对应的各个特征。
所述排除模块327,可以用于利用预设偏见语料库初步检测所述各个特征中存在的偏见特征,并从所述各个特征和所述训练数据集中分别排除所述偏见特征及其对应的结构化数据,得到排除后的各个特征和排除后的结构化数据。
所述分析模块328,可以用于根据所述排除后的结构化数据,分析所述排除后的各个特征是否为偏见特征。
在具体应用场景中,为了分析所述排除后的各个特征是否为偏见特征,所述分析模块328,可以用于若所述排除后的结构化数据存在相应的标签类别,则将所述排除后的各个特征与各个标签类别进行组合,得到多个组合结果;确定所述排除后的各个特征对应的特征值,并根据所述多个组合结果,分析在不同标签分类下各个特征值对应的第一数据量分布信息;基于所述第一数据量分布信息,判定所述排除后的各个特征是否为偏见特征。
在具体应用场景中,为了分析所述排除后的各个特征是否为偏见特征,所述分析模块328,还可以用于若所述排除后的结构化数据不存在相应的标签类别,利用预设聚类算法对所述排除后的结构化数据进行聚类处理,得到不同分类下的结构化数据;确定所述排除后的各个特征对应的特征值,并分析在不同分类下各个特征值对应的第二数据量分布信息;基于所述第二数据量分布信息,判定所述排除后的各个特征是否为偏见特征。
需要说明的是,本发明实施例提供的一种数据集质量评估装置所涉及各功能模块的其他相应描述,可以参考图1所示方法的对应描述,在此不再赘述。
基于上述如图1所示方法,相应的,本发明实施例还提供了一种计算机可读存储介质,其上存储有计算机程序,该程序被处理器执行时实现以下步骤:获取数据集中的待评估数据;分别统计所述待评估数据在多个评估维度下的属性特征;基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
基于上述如图1所示方法和如图3所示装置的实施例,本发明实施例还提供了一种计算机设备的实体结构图,如图5所示,该计算机设备包括:处理器41、存储器42、及存储在存储器42上并可在处理器上运行的计算机程序,其中存储器42和处理器41均设置在总线43上。所述处理器41执行所述程序时实现以下步骤:分别统计所述待评估数据在多个评估维度下的属性特征;基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评 估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
通过本发明的技术方案,本方明能够获取数据集中的待评估数据;并分别统计所述待评估数据在多个评估维度下的属性特征;与此同时,基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果,由此通过统计待评估数据在多个评估维度下的属性特征,能够从多个评估维度对数据集的质量进行自动评估,从而能够提高数据集质量的评估精度和评估效率,有效地保证了人工智能开发过程中数据集的安全。
显然,本领域的技术人员应该明白,上述的本发明的各模块或各步骤可以用通用的计算装置来实现,它们可以集中在单个的计算装置上,或者分布在多个计算装置所组成的网络上,可选地,它们可以用计算装置可执行的程序代码来实现,从而,可以将它们存储在存储装置中由计算装置来执行,并且在某些情况下,可以以不同于此处的顺序执行所示出或描述的步骤,或者将它们分别制作成各个集成电路模块,或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样,本发明不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已,并不用于限制本发明,对于本领域的技术人员来说,本发明可以有各种更改和变化。凡在本发明的精神和原则之内,所作的任何修改、等同替换、改进等,均应包括在本发明的保护范围之内。
Claims (10)
- 一种数据集质量评估方法,其特征在于,包括:获取数据集中的待评估数据;分别统计所述待评估数据在多个评估维度下的属性特征;基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
- 根据权利要求1所述的方法,其特征在于,所述分别统计所述待评估数据在多个评估维度下的属性特征,包括:分别统计所述待评估数据在数据规模评估维度、数据均衡性评估维度、数据准确性评估维度、数据污染评估维度、数据偏见评估维度下的属性特征;所述基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果,包括:基于所述数据规模评估维度、所述数据均衡性评估维度、所述数据准确性评估维度、所述数据污染评估维度和所述数据偏见评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述数据规模评估维度、所述数据均衡性评估维度、所述数据准确性评估维度、所述数据污染评估维度和所述数据偏见评估维度下的质量评估结果。
- 根据权利要求2所述的方法,其特征在于,所述待评估数据为训练数据集中的结构化数据,统计所述待评估数据在数据污染评估维度下的属性特征,包括:根据所述结构化数据及其对应的标签类别,利用预设插值算法拟合所述结构化数据对应的函数曲线;利用所述函数曲线对所述结构化数据进行预测,得到所述结构化数据对应的预测标签类别;基于所述预测标签类别和所述结构化数据对应的标签类别,判定所述结构化数据是否为噪声数据;基于所述数据污染评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据污染评估维度下的质量评估结果,包 括:分别统计所述噪声数据对应的第一数据量,以及结构化数据对应的第二数据量,并根据所述第一数据量和所述第二数据量,计算所述噪声数据与所述结构化数据之间的第一数据比例;若所述第一数据比例大于预设噪声数据比例,则确定所述结构化数据不满足数据污染评估标准。
- 根据权利要求2所述的方法,其特征在于,所述待评估数据为预测数据集中的非结构化数据,统计所述待评估数据在数据污染评估维度下的属性特征,包括:分别利用第一预设压缩器和第二预设压缩器对所述非结构化数据进行压缩处理,得到所述非结构化数据对应的第一压缩数据和第二压缩数据;分别对所述非结构化数据、所述第一压缩数据和所述第二压缩数据进行预测,得到所述非结构化数据对应的原始预测结果,所述第一压缩数据对应的第一预测结果,以及所述第二压缩数据对应的第二预测结果;分别计算所述原始预测结果与所述第一预测结果之间的第一差值,以及所述原始预测结果与所述第二预测结果之间的第二差值;基于所述第一差值和所述第二差值,判定所述非结构化数据是否为对抗数据;基于所述数据污染评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据在所述数据污染评估维度下的质量评估结果,包括:分别统计所述对抗数据对应的第三数据量,以及所述非结构化数据对应的第四数据量,并根据所述第三数据量和所述第四数据量,计算所述对抗数据与所述非结构化数据之间的第二数据比例;若所述第二数据比例大于预设对抗数据比例,则确定所述非结构化数据不满足数据污染评估标准。
- 根据权利要求2所述的方法,其特征在于,所述待评估数据为训练数据集中的结构化数据,统计所述待评估数据在数据偏见评估维度下的属性特征,包括:确定所述结构化数据对应的各个特征;利用预设偏见语料库初步检测所述各个特征中存在的偏见特征,并从所述各个特征和所述训练数据集中分别排除所述偏见特征及其对应的结构化数据,得到排除后的各个特征和排除后的结构化数据;根据所述排除后的结构化数据,分析所述排除后的各个特征是否为偏见特征。
- 根据权利要求5所述的方法,其特征在于,所述根据所述排除后的结构化数据,分析所述排除后的各个特征是否为偏见特征,包括:若所述排除后的结构化数据存在相应的标签类别,则将所述排除后的各个特征与各个标签类别进行组合,得到多个组合结果;确定所述排除后的各个特征对应的特征值,并根据所述多个组合结果,分析在不同标签分类下各个特征值对应的第一数据量分布信息;基于所述第一数据量分布信息,判定所述排除后的各个特征是否为偏见特征。
- 根据权利要求5所述的方法,其特征在于,所述根据所述排除后的结构化数据,分析所述排除后的各个特征是否为偏见特征,包括:若所述排除后的结构化数据不存在相应的标签类别,利用预设聚类算法对所述排除后的结构化数据进行聚类处理,得到不同分类下的结构化数据;确定所述排除后的各个特征对应的特征值,并分析在不同分类下各个特征值对应的第二数据量分布信息;基于所述第二数据量分布信息,判定所述排除后的各个特征是否为偏见特征。
- 一种数据集质量评估装置,其特征在于,包括:获取单元,用于获取数据集中的待评估数据;统计单元,用于分别统计所述待评估数据在多个评估维度下的属性特征;评估单元,用于基于所述多个评估维度下的属性特征,对所述待评估数据进行质量评估,得到所述待评估数据分别在所述多个评估维度下的质量评估结果。
- 一种计算机设备,包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,其特征在于,所述计算机程序被处理器执行时实现权利要求1至7中任一项所述的方法的步骤。
- 一种计算机可读存储介质,其上存储有计算机程序,其特征在于,所述计算机程序被处理器执行时实现权利要求1至7中任一项所述的方法的步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110999774.X | 2021-08-30 | ||
CN202110999774.XA CN113448955B (zh) | 2021-08-30 | 2021-08-30 | 数据集质量评估方法、装置、计算机设备及存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023029065A1 true WO2023029065A1 (zh) | 2023-03-09 |
Family
ID=77818805
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/117109 WO2023029065A1 (zh) | 2021-08-30 | 2021-09-08 | 数据集质量评估方法、装置、计算机设备及存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN113448955B (zh) |
WO (1) | WO2023029065A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118211040A (zh) * | 2024-05-17 | 2024-06-18 | 全拓科技(杭州)股份有限公司 | 一种用于大数据分析的数据质量评估分析方法 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118297444A (zh) * | 2024-06-06 | 2024-07-05 | 中国信息通信研究院 | 一种面向人工智能的数据集质量通用评估方法 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108764707A (zh) * | 2018-05-24 | 2018-11-06 | 国信优易数据有限公司 | 一种数据评估***以及方法 |
CN108764705A (zh) * | 2018-05-24 | 2018-11-06 | 国信优易数据有限公司 | 一种数据质量评估平台以及方法 |
CN110705607A (zh) * | 2019-09-12 | 2020-01-17 | 西安交通大学 | 一种基于循环重标注自助法的行业多标签降噪方法 |
US20200081865A1 (en) * | 2018-09-10 | 2020-03-12 | Google Llc | Rejecting Biased Data Using a Machine Learning Model |
CN112463773A (zh) * | 2019-09-06 | 2021-03-09 | 佛山市顺德区美的电热电器制造有限公司 | 数据质量确定方法及装置 |
CN112506904A (zh) * | 2020-12-02 | 2021-03-16 | 深圳市酷开网络科技股份有限公司 | 数据质量评估方法、装置、终端设备以及存储介质 |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
ATE333694T1 (de) * | 2003-01-18 | 2006-08-15 | Psytechnics Ltd | Werkzeug zur nicht invasiven bestimmung der qualität eines sprachsignals |
WO2019236560A1 (en) * | 2018-06-04 | 2019-12-12 | The Regents Of The University Of California | Pair-wise or n-way learning framework for error and quality estimation |
CN108960087A (zh) * | 2018-06-20 | 2018-12-07 | 中国科学院重庆绿色智能技术研究院 | 一种基于多维度评估标准的人脸图像质量评估方法及*** |
CN110121110B (zh) * | 2019-05-07 | 2021-05-25 | 北京奇艺世纪科技有限公司 | 视频质量评估方法、设备、视频处理设备及介质 |
CN111339215A (zh) * | 2019-05-31 | 2020-06-26 | 北京东方融信达软件技术有限公司 | 结构化数据集质量评价模型生成方法、评价方法及装置 |
CN110956224B (zh) * | 2019-08-01 | 2024-03-08 | 平安科技(深圳)有限公司 | 评估模型生成、评估数据处理方法、装置、设备及介质 |
CN111881705B (zh) * | 2019-09-29 | 2023-12-12 | 深圳数字生命研究院 | 数据处理、训练、识别方法、装置和存储介质 |
CN110956613B (zh) * | 2019-11-07 | 2023-04-07 | 成都傅立叶电子科技有限公司 | 基于图像质量的目标检测算法性能归一化评价方法及*** |
CN111523785A (zh) * | 2020-04-16 | 2020-08-11 | 三峡大学 | 一种基于生成对抗网络的电力***动态安全评估方法 |
CN111639850A (zh) * | 2020-05-27 | 2020-09-08 | 中国电力科学研究院有限公司 | 多源异构数据的质量评估方法与*** |
CN111652381A (zh) * | 2020-06-04 | 2020-09-11 | 深圳前海微众银行股份有限公司 | 数据集贡献度评估方法、装置、设备及可读存储介质 |
CN112465041B (zh) * | 2020-12-01 | 2024-01-05 | 大连海事大学 | 一种基于层次分析法的ais数据质量评估方法 |
CN113254599B (zh) * | 2021-06-28 | 2021-10-08 | 浙江大学 | 一种基于半监督学习的多标签微博文本分类方法 |
-
2021
- 2021-08-30 CN CN202110999774.XA patent/CN113448955B/zh active Active
- 2021-09-08 WO PCT/CN2021/117109 patent/WO2023029065A1/zh unknown
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108764707A (zh) * | 2018-05-24 | 2018-11-06 | 国信优易数据有限公司 | 一种数据评估***以及方法 |
CN108764705A (zh) * | 2018-05-24 | 2018-11-06 | 国信优易数据有限公司 | 一种数据质量评估平台以及方法 |
US20200081865A1 (en) * | 2018-09-10 | 2020-03-12 | Google Llc | Rejecting Biased Data Using a Machine Learning Model |
CN112463773A (zh) * | 2019-09-06 | 2021-03-09 | 佛山市顺德区美的电热电器制造有限公司 | 数据质量确定方法及装置 |
CN110705607A (zh) * | 2019-09-12 | 2020-01-17 | 西安交通大学 | 一种基于循环重标注自助法的行业多标签降噪方法 |
CN112506904A (zh) * | 2020-12-02 | 2021-03-16 | 深圳市酷开网络科技股份有限公司 | 数据质量评估方法、装置、终端设备以及存储介质 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118211040A (zh) * | 2024-05-17 | 2024-06-18 | 全拓科技(杭州)股份有限公司 | 一种用于大数据分析的数据质量评估分析方法 |
Also Published As
Publication number | Publication date |
---|---|
CN113448955B (zh) | 2021-12-07 |
CN113448955A (zh) | 2021-09-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107633265B (zh) | 用于优化信用评估模型的数据处理方法及装置 | |
KR102061987B1 (ko) | 위험 평가 방법 및 시스템 | |
WO2019214248A1 (zh) | 一种风险评估方法、装置、终端设备及存储介质 | |
AU2018101946A4 (en) | Geographical multivariate flow data spatio-temporal autocorrelation analysis method based on cellular automaton | |
WO2019184557A1 (zh) | 定位根因告警的方法、装置和计算机可读存储介质 | |
Kočišová et al. | Discriminant analysis as a tool for forecasting company's financial health | |
WO2023029065A1 (zh) | 数据集质量评估方法、装置、计算机设备及存储介质 | |
TW201734837A (zh) | 一種多重抽樣模型訓練方法及裝置 | |
CN107679734A (zh) | 一种用于无标签数据分类预测的方法和*** | |
WO2022199185A1 (zh) | 用户操作检测方法及程序产品 | |
CN110348516B (zh) | 数据处理方法、装置、存储介质及电子设备 | |
KR20160104064A (ko) | 복잡한 양자 또는 다자 상대방 관계를 탐색하기 위해 이용되는 다차원 재귀적 학습 과정 및 시스템 | |
EP4141693A1 (en) | Method and device for obtaining a generated dataset with a predetermined bias for evaluating algorithmic fairness of a machine learning model | |
WO2018036402A1 (zh) | 模型中关键变量的探测方法及装置 | |
CN114785616A (zh) | 数据风险检测方法、装置、计算机设备及存储介质 | |
CN115545103A (zh) | 异常数据识别、标签识别方法和异常数据识别装置 | |
CN114448657B (zh) | 一种配电通信网络安全态势感知与异常入侵检测方法 | |
CN108629506A (zh) | 风控模型的建模方法、装置、计算机设备和存储介质 | |
CN111815442B (zh) | 一种链接预测的方法、装置和电子设备 | |
CN110570301B (zh) | 风险识别方法、装置、设备及介质 | |
CN114626940A (zh) | 数据分析方法、装置及电子设备 | |
Wang et al. | A knowledge discovery case study of software quality prediction: Isbsg database | |
CN116484230B (zh) | 识别异常业务数据的方法及ai数字人的训练方法 | |
TWI634508B (zh) | 使用多維度評級制判定實體未來商業存活力之系統和方法 | |
CN112884167B (zh) | 一种基于机器学习的多指标异常检测方法及其应用*** |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21955596 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |