WO2020220810A1 - Data fusion method and apparatus - Google Patents
Data fusion method and apparatus Download PDFInfo
- Publication number
- WO2020220810A1 WO2020220810A1 PCT/CN2020/076895 CN2020076895W WO2020220810A1 WO 2020220810 A1 WO2020220810 A1 WO 2020220810A1 CN 2020076895 W CN2020076895 W CN 2020076895W WO 2020220810 A1 WO2020220810 A1 WO 2020220810A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- fusion
- data table
- feature
- associated data
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
Definitions
- the present invention relates to the field of computer technology, in particular to a method and device for data fusion.
- big data analysis has been applied to analysis in various fields.
- the data used in big data analysis comes from multiple heterogeneous data sources. Due to the phased, technical, and other economic and human factors of the construction and implementation of data management systems in various business systems, a large number of business data using different storage methods have been accumulated, and the data management systems used are also very different. From the file database to the complex network database, they constitute heterogeneous data sources.
- the embodiments of the present invention provide a data fusion method and device, which can collect and efficiently process data from different data sources, and perform safe and rapid fusion of data with different structures, in order to construct a unified fusion information scoring system Provide data of the same standard and format.
- a data fusion method is provided.
- a fusion model is established based on the gradient.
- filtering and merging the original data tables of each data source to obtain the associated data table of each data source includes:
- the method before performing feature analysis on the fields in the associated data table, the method further includes:
- performing feature analysis on the fields in the associated data table, and selecting features to be used includes:
- extracting gradients from each of the associated data tables according to the dimension label and the feature to be used includes:
- the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
- the method further includes:
- a data fusion device is provided.
- the merging module is used to filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table;
- the analysis module is used to perform feature analysis on the fields in the associated data table and select features to be used;
- An extraction module configured to extract gradients from each of the associated data tables according to the dimension label and the feature to be used
- the modeling module is used to establish a fusion model based on the gradient.
- the merging module is also used for:
- the merging module is further used for:
- the analysis module is also used for:
- the extraction module is also used for:
- the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
- the device further includes a calculation module for:
- a data fusion electronic device is provided.
- a data fusion electronic device includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are used by the one or more processors Execution, so that the one or more processors implement a data fusion method in an embodiment of the present invention.
- a computer-readable storage medium is provided.
- a computer-readable storage medium has a computer program stored thereon, and when the program is executed by a processor, a data fusion method according to an embodiment of the present invention is implemented.
- An embodiment of the foregoing invention has the following advantages or beneficial effects: because the original data tables of each data source are filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields in the associated data table and select the features to be used; extract gradients from each associated data table according to the dimension label and the features to be used; build a fusion model based on the gradient, and build a unified and safe
- the fusion information scoring system overcomes the lack of a unified and safe data fusion method; in the process of fusing data from heterogeneous data sources, there is a security problem of information leakage; and it cannot provide for the construction of a unified fusion information scoring system
- Figure 1 is a schematic diagram of the main steps of a data fusion method according to an embodiment of the present invention
- FIG. 2 is a schematic diagram of gradient extraction of a data fusion method according to an embodiment of the present invention
- FIG. 3 is a schematic diagram of the main flow of a data fusion method according to a reference embodiment of the present invention.
- Fig. 4 is a schematic diagram of main modules of a data fusion device according to an embodiment of the present invention.
- FIG. 5 is an exemplary system architecture diagram to which the embodiment of the present invention can be applied.
- Fig. 6 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
- the existing data fusion method in terms of data processing, is to collect data from multiple systems or platforms and import it into the same database, perform data processing uniformly, and then import the processed data into the scoring model.
- data analysis the overall data is analyzed, and then the target's score in each dimension is obtained according to the target information.
- the data fusion method proposed in the embodiment of the present invention can realize the collection and efficient processing of data from different data sources, can safely and quickly integrate data with different structures, and perform qualitative and quantitative analysis; at the same time, it can also be applied based on distributed
- the machine learning model of parallel computing reads and analyzes massive data in real time, screens for abnormal data, generates fusion information (ie analysis report) for the target, and generates a fusion score for each target based on the collected, processed, and analyzed data.
- Fig. 1 is a schematic diagram of the main steps of a data fusion method according to an embodiment of the present invention.
- the data fusion method of the embodiment of the present invention mainly includes the following steps:
- Step S101 Filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table.
- the original data table of each data source can be preprocessed separately by a third party, and the original data table of each data source can be independently preprocessed by each data source by issuing tasks to each data source.
- uniform standard screening for the original data tables of each data source the same standard data can be provided for building a unified fusion information scoring system.
- each field in the associated data table may have multiple dimension labels.
- step S101 can be implemented by the following steps: collect the original data table from various data sources; analyze the sample number, field number and null value number of the original data table; The original data table filters out the basic data table; based on the primary key of the basic data table, the basic data table of the same data source is merged into a related data table.
- the preprocessing of the original data table of each data source mainly includes data collection, data exploration, data screening, and data merging.
- Each original data table may include multiple samples composed of various fields, and each sample corresponds to a target (the target can be a certain person or something, etc.). Because the sample size or the number of fields in some original data tables is small, or there are many blank value fields, which cannot meet the requirements of subsequent analysis, the original data tables need to be filtered. Multiple basic data tables filtered from the same data source are merged into one associated data table to facilitate subsequent analysis and processing.
- the primary key of the basic data table can be the identification of the target, such as user name, address, or number.
- step S101 may further include the following steps: removing samples in the associated data table that contain null value fields or the number of null value fields exceeds the null field threshold; using box plots and field meanings to remove Abnormal fields; use the average, mode or predicted value to supplement the null field; and discretize the associated data table.
- the preprocessing of the original data table of each data source may also include data cleaning, removal of abnormal values, supplementation of missing values, discretization of variables, and the like.
- Data cleaning refers to removing some samples from the associated data table that cannot meet the requirements of subsequent analysis.
- the removed samples may contain null fields (that is, some key fields are null) or the number of null fields exceeds the null field threshold.
- the empty field threshold can be determined according to actual needs or historical data.
- the box-and-plot combined with the meaning of the field can remove the fields that have no practical meaning in the non-null fields.
- Boxplot also called Box-whisker Plot, uses five statistics in the data: minimum, first quartile, median, and third quartile A method of describing data with the maximum value, it can also roughly see whether the data has symmetry, the degree of dispersion and other information, especially for the comparison of several samples.
- the null field can be supplemented with average, mode or predicted value to facilitate subsequent processing and analysis.
- the predicted value can be obtained by means of supervised learning (such as random forest, etc.).
- Discretization is to divide the continuous variables (fields corresponding to certain characteristics) in the associated data table into discrete variables, including equal frequency division, equidistant division, and optimal division, thus reflecting the continuous variables in different ranges of values.
- the impact of the fusion model is to divide the continuous variables (fields corresponding to certain characteristics) in the associated data table into discrete variables, including equal frequency division, equidistant division, and optimal division, thus reflecting the continuous variables in different ranges of values.
- Step S102 Perform feature analysis on the fields in the associated data table, and select features to be used.
- step S102 can be implemented by the following steps: perform feature analysis and derivation on the fields in the associated data table to obtain data features and high-dimensional nonlinear features; calculate the Pearson coefficients of the data features and high-dimensional nonlinear features respectively; The sen coefficient selects the features to be used from the data features and high-dimensional nonlinear features.
- Feature derivation is based on mining new variables that affect the fusion purpose from all data types.
- Two types of features can be derived from the fields in the associated data table, namely data features and high-dimensional nonlinear features. among them,
- Data features can include but are not limited to the following categories: count (for example, the total number of loan applications in the past 12 months), sum (for example, the amount of online purchases in the past 3 months), ratio (for example, the proportion of the loan amount of the lender and the credit line) Ratio), time difference (for example, the time since the first account opening) or volatility (for example, the standard deviation of working hours in the past 3 years), etc.
- High-dimensional nonlinear features are obtained by a supervised derivation method, which can be obtained by random forest derivation to improve the nonlinear expression ability of the fusion model.
- Random forest is a classifier containing multiple decision trees, and the output category is determined by the mode of the category output by the individual tree. For example: age and income are two variables that have a "threshold effect", that is, age and income will only affect personal credit when they are greater than a threshold.
- This threshold can be passed through the two trees in the random forest (decision tree) Find out (that is, only when the age is greater than a certain threshold, there will be a sample that is overdue, and similarly when the income is greater than a certain threshold, will there be a sample that is overdue), in addition, income and age still have "dual threshold effect" variables, that is, only when Overdue samples will only appear when the age is greater than a certain threshold and the income is greater than a certain threshold. Therefore, this threshold can be found by using a decision tree in the random forest. After finding the threshold, you can add a joint variable (whether income and age are greater than their respective thresholds, the values are 00, 01, 10, 11) based on the original two variables (income, age) of each sample. In the same way, all joint variables (ie, high-dimensional nonlinear features) with "double threshold effect" can be obtained.
- Pearson correlation coefficients also known as Pearson product-moment correlation coefficients (PPMCC or PCCs) are used to measure the correlation (linear correlation) between two variables X and Y, and their values are between -1 and 1.
- PMCC Pearson product-moment correlation coefficients
- the Pearson correlation coefficient is used to describe the correlation between data features and high-dimensional nonlinear features and corresponding fields.
- Step S103 Extract a gradient from each associated data table according to the dimension label and the feature to be used.
- the technical solution of joint encryption is adopted, that is, the gradient of the parameters of each data source is extracted, that is, the specific characteristics of the corresponding fields of the features to be used can be obtained, and the original data can be avoided to ensure the information security of each data source Sex.
- the original meaning of the gradient is a vector (vector), which means that the directional derivative of a function at the point takes the maximum value along the direction, that is, the function changes the fastest and changes along the direction (the direction of the gradient) at the point.
- the maximum rate modulo the gradient).
- step S103 can be achieved by the following steps: align the associated data tables corresponding to each data source according to the shared target information, and assign a unified identifier to the target; according to the dimension label and the to-be-used feature from each association
- the gradient of the parameters of the data source and the loss of the associated data table are extracted from the data table, and the fusion loss is calculated based on the loss of each associated data table; if the fusion loss is less than the loss threshold, the extraction is terminated; if the fusion loss is greater than or equal to the loss threshold, then
- the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be re-extracted.
- data alignment operations can be performed according to the features to be used, and the alignment operations can provide data in the same format for building a unified fusion information scoring system.
- a unified identification is assigned to the target.
- the unified identification can be unique personal data (such as ID number or mobile phone number, etc.), or it can be a unique identification set artificially to ensure that different data sources have the same unified identification The data portrays the same goal.
- Dimension labels and unused features are descriptions of fields, so each dimension label corresponds to a fixed unused feature.
- the corresponding unused feature field can be extracted from each associated data table. Encrypting the gradient can ensure that the transmitted information cannot be recovered by others, and further ensures the information security of each data source. Because the parameters of the data source may have errors, the extracted gradients are not accurate enough. You can use the loss to measure whether the gradients extracted from each data source are available.
- the loss threshold here can be determined according to actual conditions or historical data.
- Step S104 Establish a fusion model based on the gradient.
- a fusion model is established based on the gradient of the parameters of the data source, which can analyze the target and obtain the fusion score of the target.
- the method of data fusion can also be used to include: extracting target's feature information to be used from each associated data table, generating fusion information based on the feature information to be used; Calculate the fusion information with features and gradients to obtain the fusion score of the target; and when the fusion score of the target changes, perform iterative calculation on the feature information to be used to obtain the influence weight of the feature to the target.
- the associated data table of each data source can be used to complete the extraction of target corresponding data (that is, extract the target feature information to be used), thereby generating fusion information.
- target corresponding data that is, extract the target feature information to be used
- For the calculation of the target's fusion score it can be obtained by calculating the to-be-used features and corresponding fields of the fusion information according to the to-be-used features and gradients.
- certain information of the target may change in certain dimensions, leading to changes in its integration score.
- the user’s personal credit may vary with time or space (such as location, industry).
- the machine learning model based on distributed parallel computing can be used to read and analyze relevant data (that is, the feature information to be used) in real time, and perform iterative calculations to obtain the target
- relevant data that is, the feature information to be used
- h the coefficient
- the coefficient can be obtained from the training data.
- the function is fitted, and the obtained parameter w is the influence weight. It should be noted that the larger the value of the influence weight, the greater the influence.
- the influence weight can also be calculated using other existing methods, which will not be repeated here.
- the original data table of each data source is filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields in the associated data table and select the features to be used; extract gradients from each associated data table according to the dimension label and the features to be used; build a fusion model based on the gradient, and build a unified and safe
- the fusion information scoring system overcomes the lack of a unified and safe data fusion method; in the process of fusing data from heterogeneous data sources, there is a security problem of information leakage; and it cannot provide for the construction of a unified fusion information scoring system
- Fig. 2 is a schematic diagram of extracting gradients of features to be used in a data fusion method according to an embodiment of the present invention.
- the integration model is constructed by integrating the data of the A department platform and the B department platform as an example.
- platform C is an independent computing platform independent of each department and is responsible for task scheduling of each platform; department A platform and department B platform respectively store two standby features in a certain dimension.
- the specific process of extracting the gradient of the parameters of the A department platform and the B department platform is as follows:
- the credit score calculation model it is necessary to calculate the score of the sample in this dimension based on all the information of the sample's identity information dimension.
- the calculation model weight iterative process only the gradient and the parameters of the A department platform are needed.
- the gradient of the parameters of the B department platform Therefore, first, the A department platform and the B department platform respectively use the gradient calculation formula to train to obtain the gradient and loss of their respective parameters. Then the gradient and loss are encrypted and sent to the platform C, and finally the platform C Decrypt the gradient and loss sent by the A department platform and the B department platform, and calculate the fusion loss based on the loss of the A department platform and the B department platform. If the fusion loss is less than the threshold, the training will be terminated; otherwise, the gradient and loss will be issued to The A department platform and the B department platform continue to train.
- the encrypted gradient is transmitted between the platform of department A and department B, between platform C and department A, and between platform C and department B.
- the respective information is unrecoverable, it guarantees The security of information on each platform.
- the gradient represents the derivative of the parameters of the model in each platform, and is an intermediate variable for updating the parameters.
- the loss of each platform is a function of the corresponding variable. The gradient and loss can be calculated by using existing calculation formulas, which are not repeated in the embodiment of the present invention.
- the data fusion method of the embodiment of the present invention can automatically store the data of various government departments and various Internet platforms in the cloud platform, and efficiently process (including: abnormal value processing, missing value Supplement, discretization processing), qualitative and quantitative analysis, and safe and fast fusion of data from multiple departments; at the same time, it can use the machine learning model based on distributed parallel computing to read and analyze massive credit data in real time, and screen for abnormal data.
- the personal credit data analysis report is automatically generated, submitted to the user, and the collected, processed, and analyzed data is input into the credit score cloud model (ie, the fusion model), and the credit score is output.
- the main process of a data fusion method that can refer to the embodiment of the present invention mainly includes the following three steps;
- the first step data preprocessing
- Preprocessing mainly includes three links: data collection, data exploration, and data processing;
- the data collection stage it is mainly to store (synchronously transmit) the data of various government departments and Internet platform data to their respective cloud platforms. First, each department determines the data tables that need to be synchronized to the cloud platform, and then synchronizes these tables from the local production environment to the cloud platform.
- the data processing stage mainly includes data cleaning, removal of outliers, supplementation of missing values, discretization of variables, etc.; first, data cleaning is performed on the associated data tables of each platform, and the value of the field is empty or null (empty Value) samples whose number exceeds the empty field threshold are eliminated. Then there is the elimination of outliers and the supplement of missing values.
- the method of elimination is to remove by combining the box plot with the specific physical meaning of the field; for the supplement of missing values, if the number of missing values is small, the average and mode are used.
- Feature derivation is based on mining new variables (features) that affect personal credit from various data types in various departments.
- Data features mainly include the following categories:
- Ratio the ratio of the loan amount of the lender to the credit line
- Time difference the time since the first account opening
- the other is to use a supervised derivation method, using random forest algorithm to derive high-dimensional nonlinear features, and improve the nonlinear expression ability of the credit scoring cloud model.
- Feature screening is to analyze the influence of existing variables on personal credit from massive data, and calculate the degree of correlation between each variable and personal credit by using machine learning algorithms (the degree of correlation can be calculated by calculating the relationship between each variable and label Pearson correlation coefficient).
- common standards include chi-square test, information gain, Gini index and correlation coefficient.
- Data fusion is the use of joint encryption technology solutions in the process of extracting data from the training model to ensure the security of information in various departments and achieve a comprehensive and multi-angle characterization of personal credit;
- a unified identifier for example: ID card number, mobile phone number.
- the corresponding fields are extracted from each platform.
- the gradient of the features to be used is extracted from each cloud platform, and a fusion model is established based on the gradient of the features to be used.
- the established fusion model can calculate the personal credit score under a certain standard. After completing the collection, processing, and analysis of personal credit data, the credit scoring model (ie, fusion model) will output a credit score (ie, fusion score) according to the user's credit information (ie, fusion information) in various dimensions. After integrating the credit information, the corresponding credit report can be returned to the user, including credit score analysis and key events affecting credit score changes.
- the user’s credit score will be analyzed from the time dimension, and from the region and industry from the spatial dimension; in the analysis of key events affecting personal credit score, the system will The platform reads and analyzes massive credit data in real time through a machine learning model based on distributed parallel computing, and obtains the weight of the user's various dimensions of credit data's influence on the credit score, and obtains the key events that cause the user's credit score to change according to the influence weight.
- scenarios involving multi-institutional data in the modeling process can use the data fusion method of the embodiment of the present invention, such as: business hall location (operator data, website data), customer recommendation (bank or website), etc.
- business hall location operator data, website data
- customer recommendation bank or website
- the embodiments of the present invention will not be listed one by one.
- Fig. 4 is a schematic diagram of main modules of a data fusion device according to an embodiment of the present invention.
- the data fusion device 400 of the embodiment of the present invention includes: a merging module 401, an analysis module 402, an extraction module 403, and a modeling module 404.
- the merging module 401 is configured to filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table;
- the analysis module 402 is configured to perform feature analysis on the fields in the associated data table and select features to be used;
- An extraction module 403, configured to extract gradients from each of the associated data tables according to the dimension label and the feature to be used;
- the modeling module 404 is configured to establish a fusion model based on the gradient.
- the merging module 401 is further configured to:
- the merging module 401 is further configured to:
- the analysis module 402 is further configured to:
- the extraction module 403 is also used to:
- the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
- the data fusion device 400 further includes a calculation module (not shown in the figure) for:
- the original data table of each data source is filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields in the associated data table and select the features to be used; extract gradients from each associated data table according to the dimension label and the features to be used; build a fusion model based on the gradient, and build a unified and safe
- the fusion information scoring system overcomes the lack of a unified and safe data fusion method; in the process of fusing data from heterogeneous data sources, there is a security problem of information leakage; and it cannot provide for the construction of a unified fusion information scoring system
- FIG. 5 shows an exemplary system architecture 500 of the data fusion method or data fusion device to which the embodiment of the present invention can be applied.
- the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505.
- the network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505.
- the network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.
- the user can use the terminal devices 501, 502, 503 to interact with the server 505 via the network 504 to receive or send messages and so on.
- Various communication client applications such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social platform software, may be installed on the terminal devices 501, 502, and 503.
- the terminal devices 501, 502, and 503 may be various electronic devices that have a display screen and support web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and so on.
- the server 505 may be a server that provides various services, for example, a background management server that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503.
- the background management server can analyze and process the received product information query request and other data, and feed back the processing result (for example, target push information, product information) to the terminal device.
- the data fusion method provided in the embodiment of the present invention is generally executed by the server 505, and accordingly, the data fusion device is generally set in the server 505.
- terminal devices, networks, and servers in FIG. 5 are merely illustrative. According to implementation needs, there can be any number of terminal devices, networks and servers.
- FIG. 6 shows a schematic structural diagram of a computer system 600 suitable for implementing the terminal device of the embodiment of the present invention.
- the terminal device shown in FIG. 6 is only an example, and should not bring any limitation to the function and application scope of the embodiment of the present invention.
- the computer system 600 includes a central processing unit (CPU) 601, which can be based on a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603 And perform various appropriate actions and processing.
- the RAM 603 also stores various programs and data required for the operation of the system 600.
- the CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604.
- An input/output (I/O) interface 605 is also connected to the bus 604.
- the following components are connected to the I/O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and speakers, etc.; a storage part 608 including a hard disk, etc. ; And a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet.
- the driver 610 is also connected to the I/O interface 605 as needed.
- a removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that the computer program read from it is installed into the storage part 608 as needed.
- the process described above with reference to the flowchart can be implemented as a computer software program.
- the disclosed embodiments of the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart.
- the computer program may be downloaded and installed from the network through the communication part 609, and/or installed from the removable medium 611.
- the central processing unit (CPU) 601 the above-mentioned functions defined in the system of the present invention are executed.
- the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two.
- the computer-readable storage medium may be, for example, but not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
- the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
- a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
- the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
- the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
- the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
- each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the above-mentioned module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions.
- the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved.
- each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be It is realized by a combination of dedicated hardware and computer instructions.
- the modules involved in the described embodiments of the present invention can be implemented in software or hardware.
- the described modules can also be provided in the processor, for example, it can be described as: a processor includes a merging module, an analysis module, an extraction module, and a modeling module.
- a processor includes a merging module, an analysis module, an extraction module, and a modeling module.
- the names of these modules do not constitute a limitation on the module itself under certain circumstances.
- the modeling module can also be described as a “module that builds a fusion model based on the gradient”.
- the present invention also provides a computer-readable medium.
- the computer-readable medium may be included in the device described in the foregoing embodiment; or it may exist alone without being assembled into the device.
- the above-mentioned computer-readable medium carries one or more programs.
- the device When the above-mentioned one or more programs are executed by a device, the device includes: Step S101: filtering and merging the original data tables of each data source to obtain each The associated data table of the data source, and add dimension labels to the fields in the associated data table; Step S102: Perform feature analysis on the fields in the associated data table, and select the features to be used; Step S103: According to the dimension labels and features to be used The gradient is extracted from the associated data table; step S104: a fusion model is established based on the gradient.
- the original data table of each data source is filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields of, and select the features to be used; extract gradients from each associated data table according to the dimension labels and features to be used; technical means to establish a fusion model based on the gradient, and build a unified and secure fusion information scoring system through the fusion model.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Disclosed is a data fusion method and apparatus, relating to the field of computer technology. A particular embodiment of said method comprises: screening and merging original data tables of data sources, so as to obtain an association data table of each data source, and adding a dimension tag to fields in the association data table; performing feature analysis on the fields in the association data table, and selecting features to be used; extracting gradients from the association data tables according to the dimension tag and the features to be used; and establishing a fusion model on the basis of the gradients. The embodiment can collect and efficiently process data of different data sources, securely and quickly fuse data of different structures, and provide data of the same standard and format for constructing a unified fusion information scoring system.
Description
本发明涉及计算机技术领域,尤其涉及一种数据融合的方法和装置。The present invention relates to the field of computer technology, in particular to a method and device for data fusion.
目前大数据分析已应用到各个领域的分析中,通常大数据分析所使用的数据来自于多个异构数据源。由于各业务***建设和实施数据管理***的阶段性、技术性以及其它经济和人为因素等因素影响,导致积累了大量采用不同存储方式的业务数据,且采用的数据管理***也大不相同,从简单的文件数据库到复杂的网络数据库,它们构成了异构数据源。At present, big data analysis has been applied to analysis in various fields. Usually, the data used in big data analysis comes from multiple heterogeneous data sources. Due to the phased, technical, and other economic and human factors of the construction and implementation of data management systems in various business systems, a large number of business data using different storage methods have been accumulated, and the data management systems used are also very different. From the file database to the complex network database, they constitute heterogeneous data sources.
在社会信用体系建设中,个人的信用评分问题一直是关键所在,通过联合各政府部门的政务数据和互联网数据,实现对失信行为的协同监管和对个人信用状况的全方面评定,对于改善各行业、各部门、各地区的信用环境,保证社会诚信和信用体系顺利建成且能有效运行具有重要作用;In the construction of the social credit system, the issue of personal credit scoring has always been the key point. By combining government affairs data and Internet data from various government departments, it is possible to realize coordinated supervision of untrustworthy behaviors and comprehensive evaluation of personal credit status, which is useful for improving various industries , The credit environment of all departments and regions has an important role in ensuring that the social integrity and credit system is successfully established and can operate effectively;
在对个人信用进行评分的过程中,由于涉及个人信用的数据来源于各个政府部门、互联网各个平台,这些数据包含不同的维度,具有不同的数据结构、含有大量重复、缺失、异常数据。如何将这些数据进行安全的整合和同步是实现个人信用评分***的关键所在。In the process of scoring personal credit, because the data related to personal credit comes from various government departments and various Internet platforms, these data contain different dimensions, have different data structures, and contain a lot of duplication, missing, and abnormal data. How to integrate and synchronize these data safely is the key to the realization of a personal credit scoring system.
此外,对于某个物品、某件事或某类事物的分析,通常也可能需要对来自异构数据源的相关数据进行全面分析。In addition, for the analysis of a certain item, a certain thing, or a certain type of things, a comprehensive analysis of related data from heterogeneous data sources may usually be required.
在实现本发明过程中,发明人发现现有技术中至少存在如下问题:In the process of implementing the present invention, the inventor found that at least the following problems exist in the prior art:
1.缺少统一的、安全的数据融合方法;1. Lack of a unified and secure data fusion method;
2.在对来自异构数据源的数据进行融合过程中,存在信息泄露的安全问题;2. In the process of fusing data from heterogeneous data sources, there are security issues of information leakage;
3.无法为构建统一的融合信息评分体系提供相同标准和格式的数 据。3. Unable to provide data of the same standard and format for building a unified fusion information scoring system.
发明内容Summary of the invention
有鉴于此,本发明实施例提供一种数据融合的方法和装置,能够对不同数据源的数据进行收集和高效处理,对不同结构的数据进行安全快速的融合,为构建统一的融合信息评分体系提供相同标准和格式的数据。In view of this, the embodiments of the present invention provide a data fusion method and device, which can collect and efficiently process data from different data sources, and perform safe and rapid fusion of data with different structures, in order to construct a unified fusion information scoring system Provide data of the same standard and format.
为实现上述目的,根据本发明实施例的一个方面,提供了一种数据融合的方法。To achieve the foregoing objective, according to an aspect of the embodiments of the present invention, a data fusion method is provided.
本发明实施例的一种数据融合的方法包括:A data fusion method in an embodiment of the present invention includes:
对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为所述关联数据表中的字段添加维度标签;Filtering and merging the original data tables of each data source to obtain the associated data table of each data source, and adding dimension labels to the fields in the associated data table;
对所述关联数据表中的字段进行特征分析,选取待用特征;Perform feature analysis on the fields in the associated data table, and select features to be used;
根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度;Extract a gradient from each of the associated data tables according to the dimension label and the feature to be used;
基于所述梯度建立融合模型。A fusion model is established based on the gradient.
可选地,对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表包括:Optionally, filtering and merging the original data tables of each data source to obtain the associated data table of each data source includes:
从各个数据源收集原始数据表;Collect raw data tables from various data sources;
分析所述原始数据表的样本数量、字段数量和空值数量;Analyze the number of samples, the number of fields, and the number of null values of the original data table;
基于所述样本数量、所述字段数量和所述空值数量从所述原始数据表筛选出基础数据表;Filtering out a basic data table from the original data table based on the number of samples, the number of fields, and the number of null values;
基于所述基础数据表的主键将同一数据源的所述基础数据表合并为关联数据表。Combining the basic data tables of the same data source into an associated data table based on the primary key of the basic data table.
可选地,在对所述关联数据表中的字段进行特征分析,选取待用特征之前还包括:Optionally, before performing feature analysis on the fields in the associated data table, the method further includes:
剔除所述关联数据表中含有空值字段或空值字段的数量超过空字段阈值的样本;Removing samples in the associated data table that contain null value fields or the number of null value fields exceeds the null field threshold;
利用箱线图结合字段意义的方式去除异常字段;Use box plots and field meanings to remove abnormal fields;
采用平均数、众数或预测值补充空值字段;以及Use average, mode, or predicted values to supplement empty value fields; and
对所述关联数据表进行离散化处理。Discretize the associated data table.
可选地,对所述关联数据表中的字段进行特征分析,选取待用特征包括:Optionally, performing feature analysis on the fields in the associated data table, and selecting features to be used includes:
对所述关联数据表中的字段进行特征衍生,得到数据特征和高维非线性特征;Perform feature derivation on the fields in the associated data table to obtain data features and high-dimensional nonlinear features;
分别计算所述数据特征和所述高维非线性特征的皮尔逊系数;Respectively calculating the Pearson coefficients of the data feature and the high-dimensional nonlinear feature;
基于所述皮尔逊系数从所述数据特征和所述高维非线性特征中选取待用特征。Select a feature to be used from the data feature and the high-dimensional nonlinear feature based on the Pearson coefficient.
可选地,根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度包括:Optionally, extracting gradients from each of the associated data tables according to the dimension label and the feature to be used includes:
根据所述待用特征对各个数据源对应的所述关联数据表进行对齐操作,并为目标分配统一标识;Aligning the associated data table corresponding to each data source according to the features to be used, and assigning a unified identifier to the target;
根据所述维度标签和所述待用特征从各个所述关联数据表中抽取数据源的参数的梯度和所述关联数据表的损失,并基于各个所述关联数据表的损失计算融合损失;Extract the gradient of the parameter of the data source and the loss of the associated data table from each of the associated data tables according to the dimensional label and the features to be used, and calculate a fusion loss based on the loss of each of the associated data tables;
若所述融合损失小于损失阈值,则终止抽取;If the fusion loss is less than the loss threshold, the extraction is terminated;
若所述融合损失大于或等于损失阈值,则将更新各个数据源的参数,并重新抽取数据源的参数的梯度和所述关联数据表的损失。If the fusion loss is greater than or equal to the loss threshold, the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
可选地,所述方法还包括:Optionally, the method further includes:
从各个所述关联数据表中提取目标的待用特征信息,基于所述待用特征信息生成融合信息;Extracting target feature information for use from each of the associated data tables, and generating fusion information based on the feature information for use;
将所述融合信息输入所述融合模型,以根据所述待用特征和所述梯度对所述融合信息进行计算,得到目标的融合分值;以及Inputting the fusion information into the fusion model to calculate the fusion information according to the features to be used and the gradient to obtain the fusion score of the target; and
在目标的融合分值变化时,对所述待用特征信息进行迭代计算,得到所述待用特征对目标的影响权重。When the fusion score of the target changes, iterative calculation is performed on the to-be-used feature information to obtain the influence weight of the to-be-used feature on the target.
为实现上述目的,根据本发明实施例的另一方面,提供了一种数据融合的装置。To achieve the foregoing objective, according to another aspect of the embodiments of the present invention, a data fusion device is provided.
本发明实施例的一种数据融合的装置包括:A data fusion device in an embodiment of the present invention includes:
合并模块,用于对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为所述关联数据表中的字段添加维度标签;The merging module is used to filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table;
分析模块,用于对所述关联数据表中的字段进行特征分析,选取待用特征;The analysis module is used to perform feature analysis on the fields in the associated data table and select features to be used;
抽取模块,用于根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度;An extraction module, configured to extract gradients from each of the associated data tables according to the dimension label and the feature to be used;
建模模块,用于基于所述梯度建立融合模型。The modeling module is used to establish a fusion model based on the gradient.
可选地,所述合并模块还用于:Optionally, the merging module is also used for:
从各个数据源收集原始数据表;Collect raw data tables from various data sources;
分析所述原始数据表的样本数量、字段数量和空值数量;Analyze the number of samples, the number of fields, and the number of null values of the original data table;
基于所述样本数量、所述字段数量和所述空值数量从所述原始数据表筛选出基础数据表;Filtering out a basic data table from the original data table based on the number of samples, the number of fields, and the number of null values;
基于所述基础数据表的主键将同一数据源的所述基础数据表合并为关联数据表。Combining the basic data tables of the same data source into an associated data table based on the primary key of the basic data table.
可选地,所述合并模块进一步用于:Optionally, the merging module is further used for:
剔除所述关联数据表中含有空值字段或空值字段的数量超过空字段阈值的样本;Removing samples in the associated data table that contain null value fields or the number of null value fields exceeds the null field threshold;
利用箱线图结合字段意义的方式去除异常字段;Use box plots and field meanings to remove abnormal fields;
采用平均数、众数或预测值补充空值字段;以及Use average, mode, or predicted values to supplement empty value fields; and
对所述关联数据表进行离散化处理。Discretize the associated data table.
可选地,所述分析模块还用于:Optionally, the analysis module is also used for:
对所述关联数据表中的字段进行特征衍生,得到数据特征和高维非线性特征;Perform feature derivation on the fields in the associated data table to obtain data features and high-dimensional nonlinear features;
分别计算所述数据特征和所述高维非线性特征的皮尔逊系数;Respectively calculating the Pearson coefficients of the data feature and the high-dimensional nonlinear feature;
基于所述皮尔逊系数从所述数据特征和所述高维非线性特征中选取待用特征。Select a feature to be used from the data feature and the high-dimensional nonlinear feature based on the Pearson coefficient.
可选地,所述抽取模块还用于:Optionally, the extraction module is also used for:
根据所述待用特征对各个数据源对应的所述关联数据表进行对齐操作,并为目标分配统一标识;Aligning the associated data table corresponding to each data source according to the features to be used, and assigning a unified identifier to the target;
根据所述维度标签和所述待用特征从各个所述关联数据表中抽取数据源的参数的梯度和所述关联数据表的损失,并基于各个所述关联数据表的损失计算融合损失;Extract the gradient of the parameter of the data source and the loss of the associated data table from each of the associated data tables according to the dimensional label and the features to be used, and calculate a fusion loss based on the loss of each of the associated data tables;
若所述融合损失小于损失阈值,则终止抽取;If the fusion loss is less than the loss threshold, the extraction is terminated;
若所述融合损失大于或等于损失阈值,则将更新各个数据源的参数,并重新抽取数据源的参数的梯度和所述关联数据表的损失。If the fusion loss is greater than or equal to the loss threshold, the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
可选地,所述装置还包括计算模块,用于:Optionally, the device further includes a calculation module for:
从各个所述关联数据表中提取目标的待用特征信息,基于所述待用特征信息生成融合信息;Extracting target feature information for use from each of the associated data tables, and generating fusion information based on the feature information for use;
将所述融合信息输入所述融合模型,以根据所述待用特征和所述梯度对所述融合信息进行计算,得到目标的融合分值;以及Inputting the fusion information into the fusion model to calculate the fusion information according to the features to be used and the gradient to obtain the fusion score of the target; and
在目标的融合分值变化时,对所述待用特征信息进行迭代计算,得到所述待用特征对目标的影响权重。When the fusion score of the target changes, iterative calculation is performed on the to-be-used feature information to obtain the influence weight of the to-be-used feature on the target.
为实现上述目的,根据本发明实施例的又一方面,提供了一种数据融合的电子设备。To achieve the foregoing objective, according to another aspect of the embodiments of the present invention, a data fusion electronic device is provided.
本发明实施例的一种数据融合的电子设备包括:一个或多个处理 器;存储装置,用于存储一个或多个程序,当所述一个或多个程序被所述一个或多个处理器执行,使得所述一个或多个处理器实现本发明实施例的一种数据融合的方法。A data fusion electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are used by the one or more processors Execution, so that the one or more processors implement a data fusion method in an embodiment of the present invention.
为实现上述目的,根据本发明实施例的再一方面,提供了一种计算机可读存储介质。To achieve the foregoing objective, according to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided.
本发明实施例的一种计算机可读存储介质,其上存储有计算机程序,所述程序被处理器执行时实现本发明实施例的一种数据融合的方法。A computer-readable storage medium according to an embodiment of the present invention has a computer program stored thereon, and when the program is executed by a processor, a data fusion method according to an embodiment of the present invention is implemented.
上述发明中的一个实施例具有如下优点或有益效果:因为采用对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为关联数据表中的字段添加维度标签;对关联数据表中的字段进行特征分析,选取待用特征;根据维度标签和待用特征从各个关联数据表中抽取梯度;基于梯度建立融合模型的技术手段,通过融合模型构建统一的、安全的融合信息评分体系,所以克服了缺少统一的、安全的数据融合方法;在对来自异构数据源的数据进行融合过程中,存在信息泄露的安全问题;以及无法为构建统一的融合信息评分体系提供相同标准和格式的数据的技术问题,进而达到对不同数据源的数据进行收集和高效处理,对不同结构的数据进行安全快速的融合,为构建统一的融合信息评分体系提供相同标准和格式的数据的技术效果。An embodiment of the foregoing invention has the following advantages or beneficial effects: because the original data tables of each data source are filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields in the associated data table and select the features to be used; extract gradients from each associated data table according to the dimension label and the features to be used; build a fusion model based on the gradient, and build a unified and safe The fusion information scoring system overcomes the lack of a unified and safe data fusion method; in the process of fusing data from heterogeneous data sources, there is a security problem of information leakage; and it cannot provide for the construction of a unified fusion information scoring system The technical problems of data of the same standard and format, and then achieve the collection and efficient processing of data from different data sources, the safe and rapid fusion of data of different structures, and provide data of the same standard and format for building a unified integrated information scoring system Technical effect.
上述的非惯用的可选方式所具有的进一步效果将在下文中结合具体实施方式加以说明。The further effects of the above non-conventional alternative methods will be described below in conjunction with specific implementations.
附图用于更好地理解本发明,不构成对本发明的不当限定。其中:The accompanying drawings are used to better understand the present invention, and do not constitute an improper limitation of the present invention. among them:
图1是根据本发明实施例的数据融合的方法的主要步骤的示意图;Figure 1 is a schematic diagram of the main steps of a data fusion method according to an embodiment of the present invention;
图2是根据本发明实施例的数据融合的方法的抽取梯度的示意图;2 is a schematic diagram of gradient extraction of a data fusion method according to an embodiment of the present invention;
图3是根据本发明一个可参考实施例的数据融合的方法的主要流程的示意图;3 is a schematic diagram of the main flow of a data fusion method according to a reference embodiment of the present invention;
图4是根据本发明实施例的数据融合的装置的主要模块的示意图;Fig. 4 is a schematic diagram of main modules of a data fusion device according to an embodiment of the present invention;
图5是本发明实施例可以应用于其中的示例性***架构图;FIG. 5 is an exemplary system architecture diagram to which the embodiment of the present invention can be applied;
图6是适于用来实现本发明实施例的终端设备或服务器的计算机***的结构示意图。Fig. 6 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
以下结合附图对本发明的示范性实施例做出说明,其中包括本发明实施例的各种细节以助于理解,应当将它们认为仅仅是示范性的。因此,本领域普通技术人员应当认识到,可以对这里描述的实施例做出各种改变和修改,而不会背离本发明的范围和精神。同样,为了清楚和简明,以下的描述中省略了对公知功能和结构的描述。The following describes exemplary embodiments of the present invention with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and should be regarded as merely exemplary. Therefore, those of ordinary skill in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Likewise, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
需要指出的是,在不冲突的情况下,本发明的实施例以及实施例中的技术特征可以相互结合。It should be pointed out that the embodiments of the present invention and the technical features in the embodiments can be combined with each other if there is no conflict.
现有的数据融合方法,在数据处理方面,是从多个***或平台收集数据导入到同一数据库中,统一进行数据处理,然后将处理后的数据导入到评分模型中。在数据分析方面,是对整体数据进行分析,再根据目标信息得到目标在各个维度的评分。The existing data fusion method, in terms of data processing, is to collect data from multiple systems or platforms and import it into the same database, perform data processing uniformly, and then import the processed data into the scoring model. In terms of data analysis, the overall data is analyzed, and then the target's score in each dimension is obtained according to the target information.
而多个***或平台的数据在同步、融合处理方面的成本较高、安全性低,且存在数据格式不同和通讯协议互不兼容的问题,难以集中、高效处理的问题。However, the synchronization and fusion processing of data from multiple systems or platforms has high costs and low security, and there are problems of different data formats and incompatibility of communication protocols, and it is difficult to concentrate and efficiently process.
本发明实施例提出的数据融合的方法,可以实现对不同数据源的数据进行收集和高效处理,能够对不同结构的数据进行安全快速的融合,并进行定性定量分析;同时还能够应用基于分布式并行计算的机器学习模型实时读取、分析海量数据,筛查异常数据,为目标生成融合信息(即分析报告),并根据收集、处理、分析后的数据为每个目 标生成融合分值。The data fusion method proposed in the embodiment of the present invention can realize the collection and efficient processing of data from different data sources, can safely and quickly integrate data with different structures, and perform qualitative and quantitative analysis; at the same time, it can also be applied based on distributed The machine learning model of parallel computing reads and analyzes massive data in real time, screens for abnormal data, generates fusion information (ie analysis report) for the target, and generates a fusion score for each target based on the collected, processed, and analyzed data.
图1是根据本发明实施例的数据融合的方法的主要步骤的示意图。Fig. 1 is a schematic diagram of the main steps of a data fusion method according to an embodiment of the present invention.
如图1所示,本发明实施例的数据融合的方法主要包括以下步骤:As shown in Figure 1, the data fusion method of the embodiment of the present invention mainly includes the following steps:
步骤S101:对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为关联数据表中的字段添加维度标签。Step S101: Filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table.
由于最终建立融合模型所需的数据来自于各个数据源,而各个数据源的数据结构、数据规模或数据质量等可能差异巨大,因此需要对各个数据源的原始数据表先进行预处理,以筛选出可用的数据,并将同一数据源的所有原始数据表合并为一个关联数据表。在此阶段,可以通过第三方分别对各个数据源的原始数据表进行预处理,还可以通过向各个数据源分别下发任务的方式,使各个数据源独立对自身的原始数据表进行预处理。通过对各个数据源的原始数据表采用统一标准的筛选,可以为构建统一的融合信息评分体系提供相同标准的数据。Since the data required for the final establishment of the fusion model comes from various data sources, and the data structure, data scale, or data quality of each data source may vary greatly, it is necessary to preprocess the original data table of each data source to filter Output the available data, and merge all the original data tables of the same data source into one related data table. At this stage, the original data table of each data source can be preprocessed separately by a third party, and the original data table of each data source can be independently preprocessed by each data source by issuing tasks to each data source. Through the use of uniform standard screening for the original data tables of each data source, the same standard data can be provided for building a unified fusion information scoring system.
为了提高对融合信息的全面刻画能力,在得到各个平台的关联数据表之后,将关联数据表中的所有字段分成若干个维度并添加维度标签,例如用于分析个人信息的数据,可以将所有字段划分为:身份特征、消费能力、信用历史、行为偏好和负债情况五个维度。需要注意的是,关联数据表中的每个字段可能具有多个维度标签。In order to improve the ability to comprehensively describe the fusion information, after obtaining the associated data table of each platform, divide all the fields in the associated data table into several dimensions and add dimension labels. For example, for data analysis of personal information, all fields can be It is divided into five dimensions: identity characteristics, consumption power, credit history, behavior preference and debt situation. It should be noted that each field in the associated data table may have multiple dimension labels.
在本发明实施例中,步骤S101可以通过以下步骤实现:从各个数据源收集原始数据表;分析原始数据表的样本数量、字段数量和空值数量;基于样本数量、字段数量和空值数量从原始数据表筛选出基础数据表;基于基础数据表的主键将同一数据源的基础数据表合并为关联数据表。In the embodiment of the present invention, step S101 can be implemented by the following steps: collect the original data table from various data sources; analyze the sample number, field number and null value number of the original data table; The original data table filters out the basic data table; based on the primary key of the basic data table, the basic data table of the same data source is merged into a related data table.
本发明实施例的数据融合的方法中,对各个数据源的原始数据表的预处理主要包括数据收集、数据探查、数据筛选和数据合并等。每个原始数据表中可能包括多个由各类字段组成的样本,每个样本对应一个目标(该目标可以是某个人或某件事等等)。由于部分原始数据 表中的样本数量或字段数量较少、或空值字段较多,而不能满足后续分析要求,因此需要对原始数据表进行筛选。对于从同一数据源筛选出的多个基础数据表合并为一个关联数据表,以便于后续的分析处理。基础数据表的主键可以是目标的标识,例如用户名、地址或号码等。In the data fusion method of the embodiment of the present invention, the preprocessing of the original data table of each data source mainly includes data collection, data exploration, data screening, and data merging. Each original data table may include multiple samples composed of various fields, and each sample corresponds to a target (the target can be a certain person or something, etc.). Because the sample size or the number of fields in some original data tables is small, or there are many blank value fields, which cannot meet the requirements of subsequent analysis, the original data tables need to be filtered. Multiple basic data tables filtered from the same data source are merged into one associated data table to facilitate subsequent analysis and processing. The primary key of the basic data table can be the identification of the target, such as user name, address, or number.
在本发明实施例中,对于步骤S101的实现还可以包括以下步骤:剔除关联数据表中含有空值字段或空值字段的数量超过空字段阈值的样本;利用箱线图结合字段意义的方式去除异常字段;采用平均数、众数或预测值补充空值字段;以及对关联数据表进行离散化处理。In the embodiment of the present invention, the implementation of step S101 may further include the following steps: removing samples in the associated data table that contain null value fields or the number of null value fields exceeds the null field threshold; using box plots and field meanings to remove Abnormal fields; use the average, mode or predicted value to supplement the null field; and discretize the associated data table.
本发明实施例的数据融合的方法中,对各个数据源的原始数据表的预处理还可以包括数据清洗、异常值的剔除、缺失值的补充和变量离散化等。数据清洗即从关联数据表中剔除一部分不能满足后续分析要求的样本,被剔除的样本可能是因含有空值字段(即某些关键字段为空值)或空值字段的数量超过空字段阈值等数据质量较低的样本,该空字段阈值可以根据实际需要或历史数据确定。同时利用箱线图结合字段意义的方式可以去除非空值的字段中不具有实际意义的字段,例如根据箱线图判断一年内逾期次数20次是异常字段,且根据逻辑关系知道一年内逾期20次确为不可能。其中,箱线图(Boxplot)也称箱须图(Box-whisker Plot),是利用数据中的五个统计量:最小值、第一四分位数、中位数、第三四分位数与最大值来描述数据的一种方法,它也可以粗略地看出数据是否具有对称性,分布的分散程度等信息,特别可以用于对几个样本的比较。对于空值字段可以采用平均数、众数或预测值进行补充,以便于后续处理和分析,例如采用监督学习的方式(如随机森林等)得出预测值。离散化处理是将关联数据表中的连续变量(某些特征对应的字段)划分成离散变量,包括等频划分、等距划分、最优划分,从而体现出连续变量在不同的取值范围对融合模型的影响。In the data fusion method of the embodiment of the present invention, the preprocessing of the original data table of each data source may also include data cleaning, removal of abnormal values, supplementation of missing values, discretization of variables, and the like. Data cleaning refers to removing some samples from the associated data table that cannot meet the requirements of subsequent analysis. The removed samples may contain null fields (that is, some key fields are null) or the number of null fields exceeds the null field threshold. For samples with low data quality, the empty field threshold can be determined according to actual needs or historical data. At the same time, the box-and-plot combined with the meaning of the field can remove the fields that have no practical meaning in the non-null fields. For example, according to the box-and-plot, it is judged that 20 overdue times in a year is an abnormal field, and according to the logical relationship, it is known that 20 overdue in a year is an abnormal field. This is indeed impossible. Among them, Boxplot, also called Box-whisker Plot, uses five statistics in the data: minimum, first quartile, median, and third quartile A method of describing data with the maximum value, it can also roughly see whether the data has symmetry, the degree of dispersion and other information, especially for the comparison of several samples. The null field can be supplemented with average, mode or predicted value to facilitate subsequent processing and analysis. For example, the predicted value can be obtained by means of supervised learning (such as random forest, etc.). Discretization is to divide the continuous variables (fields corresponding to certain characteristics) in the associated data table into discrete variables, including equal frequency division, equidistant division, and optimal division, thus reflecting the continuous variables in different ranges of values. The impact of the fusion model.
步骤S102:对关联数据表中的字段进行特征分析,选取待用特征。Step S102: Perform feature analysis on the fields in the associated data table, and select features to be used.
在各个数据源的数据分别进行处理得到关联数据表后,需要对各 个数据源的数据进行特征分析,即分析关联数据表中所有字段的特征,以得到待用特征。After the data of each data source is processed separately to obtain the associated data table, it is necessary to perform feature analysis on the data of each data source, that is, to analyze the characteristics of all fields in the associated data table to obtain the features to be used.
在本发明实施例中,步骤S102可以通过以下步骤实现:对关联数据表中的字段进行特征分析衍生,得到数据特征和高维非线性特征;分别计算数据特征和高维非线性特征的皮尔逊系数;基于皮尔逊系数从数据特征和高维非线性特征中选取待用特征。In the embodiment of the present invention, step S102 can be implemented by the following steps: perform feature analysis and derivation on the fields in the associated data table to obtain data features and high-dimensional nonlinear features; calculate the Pearson coefficients of the data features and high-dimensional nonlinear features respectively; The sen coefficient selects the features to be used from the data features and high-dimensional nonlinear features.
特征衍生是根据从所有的数据类型中挖掘出影响融合目的的新变量,根据关联数据表中的字段可以衍生出两类特征,即数据特征和高维非线性特征。其中,Feature derivation is based on mining new variables that affect the fusion purpose from all data types. Two types of features can be derived from the fields in the associated data table, namely data features and high-dimensional nonlinear features. among them,
数据特征可以包括但不限于以下几类:计数(例如过去12个月内申请贷款的总次数)、求和(例如过去3个月网购金额)、比例(例如贷款人贷款金额与授信额度的占比)、时间差(例如第一次开户距今时长)或波动率(例如过去3年工作时间的标准差)等等。Data features can include but are not limited to the following categories: count (for example, the total number of loan applications in the past 12 months), sum (for example, the amount of online purchases in the past 3 months), ratio (for example, the proportion of the loan amount of the lender and the credit line) Ratio), time difference (for example, the time since the first account opening) or volatility (for example, the standard deviation of working hours in the past 3 years), etc.
高维非线性特征是一种采用有监督的衍生方式得到的,可以采用随机森林衍生得到,用于提高融合模型的非线性表达能力。随机森林是一个包含多个决策树的分类器,其输出的类别是由个别树输出的类别的众数而定。例如:年龄和收入是两个存在“阈值效应”的变量,即年龄和收入只有大于一个阈值的时候才会对个人信用有影响,这个阈值可以通过随机森林的中的两颗树(decision tree)找出(即只有当年龄大于一定阈值,才会有出现逾期的样本,同理当收入大于一定阈值才会出现逾期的样本),此外收入和年龄还是存在“双阈值效应”的变量,即只有当年龄大于一定阈值且收入大于一定阈值的时候才会有出现逾期样本,因此可以通过使用随机森林的中的一颗树(decision tree)来找出此阈值。找出阈值之后,根据原本每个样本的两个变量(收入、年龄)就可以增加联合变量(收入和年龄是否大于各自阈值,取值有00、01、10、11四个情况)。同理可以得到所有存在“双阈值效应”的联合变量(即高维非线性特征)。High-dimensional nonlinear features are obtained by a supervised derivation method, which can be obtained by random forest derivation to improve the nonlinear expression ability of the fusion model. Random forest is a classifier containing multiple decision trees, and the output category is determined by the mode of the category output by the individual tree. For example: age and income are two variables that have a "threshold effect", that is, age and income will only affect personal credit when they are greater than a threshold. This threshold can be passed through the two trees in the random forest (decision tree) Find out (that is, only when the age is greater than a certain threshold, there will be a sample that is overdue, and similarly when the income is greater than a certain threshold, will there be a sample that is overdue), in addition, income and age still have "dual threshold effect" variables, that is, only when Overdue samples will only appear when the age is greater than a certain threshold and the income is greater than a certain threshold. Therefore, this threshold can be found by using a decision tree in the random forest. After finding the threshold, you can add a joint variable (whether income and age are greater than their respective thresholds, the values are 00, 01, 10, 11) based on the original two variables (income, age) of each sample. In the same way, all joint variables (ie, high-dimensional nonlinear features) with "double threshold effect" can be obtained.
皮尔逊相关系数,又称皮尔逊积矩相关系数(PPMCC或PCCs),是用于度量两个变量X和Y之间的相关(线性相关),其值介于-1与 1之间。本发明实施例的数据融合的方法中,使用皮尔逊相关系数描述数据特征和高维非线性特征与对应字段的相关性。Pearson correlation coefficients, also known as Pearson product-moment correlation coefficients (PPMCC or PCCs), are used to measure the correlation (linear correlation) between two variables X and Y, and their values are between -1 and 1. In the data fusion method of the embodiment of the present invention, the Pearson correlation coefficient is used to describe the correlation between data features and high-dimensional nonlinear features and corresponding fields.
步骤S103:根据维度标签和待用特征从各个关联数据表中抽取梯度。Step S103: Extract a gradient from each associated data table according to the dimension label and the feature to be used.
在完成各个数据源的数据处理和特征分析后,需要对所有关联数据表的数据进行联合建模,实现对融合目的的全方位多角度刻画。在抽取数据的过程中,采用联合加密的技术方案,即抽取各个数据源的参数的梯度,即能够得到待用特征对应字段的具体特征,又能够避免暴露原始数据,确保各个数据源的信息安全性。梯度的本意是一个向量(矢量),表示某一函数在该点处的方向导数沿着该方向取得最大值,即函数在该点处沿着该方向(此梯度的方向)变化最快、变化率最大(为该梯度的模)。After completing the data processing and feature analysis of each data source, it is necessary to jointly model the data of all related data tables to achieve a comprehensive and multi-angle characterization of the fusion purpose. In the process of extracting data, the technical solution of joint encryption is adopted, that is, the gradient of the parameters of each data source is extracted, that is, the specific characteristics of the corresponding fields of the features to be used can be obtained, and the original data can be avoided to ensure the information security of each data source Sex. The original meaning of the gradient is a vector (vector), which means that the directional derivative of a function at the point takes the maximum value along the direction, that is, the function changes the fastest and changes along the direction (the direction of the gradient) at the point. The maximum rate (modulo the gradient).
在本发明实施例中,步骤S103可以通过以下步骤实现:根据共有的目标信息对各个数据源对应的关联数据表进行对齐操作,并为目标分配统一标识;根据维度标签和待用特征从各个关联数据表中抽取数据源的参数的梯度和关联数据表的损失,并基于各个关联数据表的损失计算融合损失;若融合损失小于损失阈值,则终止抽取;若融合损失大于或等于损失阈值,则将更新各个数据源的参数,并重新抽取数据源的参数的梯度和关联数据表的损失。In the embodiment of the present invention, step S103 can be achieved by the following steps: align the associated data tables corresponding to each data source according to the shared target information, and assign a unified identifier to the target; according to the dimension label and the to-be-used feature from each association The gradient of the parameters of the data source and the loss of the associated data table are extracted from the data table, and the fusion loss is calculated based on the loss of each associated data table; if the fusion loss is less than the loss threshold, the extraction is terminated; if the fusion loss is greater than or equal to the loss threshold, then The parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be re-extracted.
对于各个数据源的经过处理分析后的关联数据表,可以根据待用特征进行数据对齐操作,对齐操作能够为构建统一的融合信息评分体系提供相同格式的数据。同时为目标分配统一标识,该统一标识可以是具有唯一性的个人数据(例如身份证号码或手机号等),也可以是人为设定的唯一标志,以保证来自不同数据源而具有同一统一标识的数据刻画的是同一目标。For the associated data tables after processing and analysis of each data source, data alignment operations can be performed according to the features to be used, and the alignment operations can provide data in the same format for building a unified fusion information scoring system. At the same time, a unified identification is assigned to the target. The unified identification can be unique personal data (such as ID number or mobile phone number, etc.), or it can be a unique identification set artificially to ensure that different data sources have the same unified identification The data portrays the same goal.
维度标签和待用特征都是对字段的描述,因此每个维度标签对应有固定的待用特征,在基于各个维度建模过程中,可以从各个关联数据表中抽取相应待用特征的字段。对梯度进行加密能够保证传输的信 息的不可被他人恢复,进一步保证了各数据源的信息安全性。由于数据源的参数可能存在误差,导致抽取的梯度不够准确,可以使用损失度量从各个数据源抽取的梯度是否可用,此处的损失阈值可以根据实际情况或历史数据确定。Dimension labels and unused features are descriptions of fields, so each dimension label corresponds to a fixed unused feature. In the process of modeling based on each dimension, the corresponding unused feature field can be extracted from each associated data table. Encrypting the gradient can ensure that the transmitted information cannot be recovered by others, and further ensures the information security of each data source. Because the parameters of the data source may have errors, the extracted gradients are not accurate enough. You can use the loss to measure whether the gradients extracted from each data source are available. The loss threshold here can be determined according to actual conditions or historical data.
步骤S104:基于梯度建立融合模型。Step S104: Establish a fusion model based on the gradient.
最后以数据源的参数的梯度为基础建立融合模型,该融合模型能够对目标进行分析,得出目标的融合分值。Finally, a fusion model is established based on the gradient of the parameters of the data source, which can analyze the target and obtain the fusion score of the target.
在本发明实施例中,数据融合的方法还可以用于包括:从各个关联数据表中提取目标的待用特征信息,基于待用特征信息生成融合信息;将融合信息输入融合模型,以根据待用特征和梯度对融合信息进行计算,得到目标的融合分值;以及在目标的融合分值变化时,对待用特征信息进行迭代计算,得到待用特征对目标的影响权重。In the embodiment of the present invention, the method of data fusion can also be used to include: extracting target's feature information to be used from each associated data table, generating fusion information based on the feature information to be used; Calculate the fusion information with features and gradients to obtain the fusion score of the target; and when the fusion score of the target changes, perform iterative calculation on the feature information to be used to obtain the influence weight of the feature to the target.
利用各个数据源的关联数据表可以完成目标对应数据的提取(即提取目标的待用特征信息),从而生成融合信息。对于目标的融合分值的计算,可以根据待用特征和梯度对融合信息所具有的待用特征及对应的字段进行计算得到。此外,目标的某些信息可能会在某些维度上发生变化,导致其融合分值发生变化,例如用户的个人信用可能在随着时间维度或空间维度(如所处地区、所处行业)的变化而发生变化,在分析影响融合分值的关键事件时,可以通过基于分布式并行计算的机器学习模型实时读取、分析相关数据(即待用特征信息),并进行迭代计算,得到目标的各个待用特征对融合分值的影响权重,并根据影响权重得到导致融合分值出现变化的关键事件。具体地,可以构造出一个回归函数y=h(w*x),其中x是基本数据,y是标签(待用特征对应的字段),h是系数,系数可以通过训练数据得到,对该回归函数进行拟合,所得参数w即为影响权重。需要注意的是,影响权重的取值越大,表示影响越大。且影响权重还可以采用其它的现有方式计算得到,此处不予赘述。The associated data table of each data source can be used to complete the extraction of target corresponding data (that is, extract the target feature information to be used), thereby generating fusion information. For the calculation of the target's fusion score, it can be obtained by calculating the to-be-used features and corresponding fields of the fusion information according to the to-be-used features and gradients. In addition, certain information of the target may change in certain dimensions, leading to changes in its integration score. For example, the user’s personal credit may vary with time or space (such as location, industry). When analyzing the key events that affect the fusion score, the machine learning model based on distributed parallel computing can be used to read and analyze relevant data (that is, the feature information to be used) in real time, and perform iterative calculations to obtain the target The influence weight of each feature to be used on the fusion score, and the key event that causes the fusion score to change according to the influence weight. Specifically, a regression function y=h(w*x) can be constructed, where x is the basic data, y is the label (the field corresponding to the feature to be used), and h is the coefficient. The coefficient can be obtained from the training data. The function is fitted, and the obtained parameter w is the influence weight. It should be noted that the larger the value of the influence weight, the greater the influence. In addition, the influence weight can also be calculated using other existing methods, which will not be repeated here.
根据本发明实施例的数据融合的方法可以看出,因为采用对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为关联数据表中的字段添加维度标签;对关联数据表中的字段进行特征分析,选取待用特征;根据维度标签和待用特征从各个关联数据表中抽取梯度;基于梯度建立融合模型的技术手段,通过融合模型构建统一的、安全的融合信息评分体系,所以克服了缺少统一的、安全的数据融合方法;在对来自异构数据源的数据进行融合过程中,存在信息泄露的安全问题;以及无法为构建统一的融合信息评分体系提供相同标准和格式的数据的技术问题,进而达到对不同数据源的数据进行收集和高效处理,对不同结构的数据进行安全快速的融合,为构建统一的融合信息评分体系提供相同标准和格式的数据的技术效果。According to the data fusion method of the embodiment of the present invention, it can be seen that the original data table of each data source is filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields in the associated data table and select the features to be used; extract gradients from each associated data table according to the dimension label and the features to be used; build a fusion model based on the gradient, and build a unified and safe The fusion information scoring system overcomes the lack of a unified and safe data fusion method; in the process of fusing data from heterogeneous data sources, there is a security problem of information leakage; and it cannot provide for the construction of a unified fusion information scoring system The technical problems of data of the same standard and format, and then achieve the collection and efficient processing of data from different data sources, the safe and rapid fusion of data of different structures, and provide data of the same standard and format for building a unified integrated information scoring system Technical effect.
图2是根据本发明实施例的数据融合的方法的抽取待用特征的梯度的示意图。Fig. 2 is a schematic diagram of extracting gradients of features to be used in a data fusion method according to an embodiment of the present invention.
如图2所示,以融合A部门平台和B部门平台的数据来构建融合模型为例。假设,平台C是独立于各个部门的独立计算平台,负责各个平台的任务调度;A部门平台和B部门平台分别存储了某一维度的两种待用特征。则抽取A部门平台和B部门平台的参数的梯度的具体过程如下:As shown in Figure 2, the integration model is constructed by integrating the data of the A department platform and the B department platform as an example. Assume that platform C is an independent computing platform independent of each department and is responsible for task scheduling of each platform; department A platform and department B platform respectively store two standby features in a certain dimension. Then the specific process of extracting the gradient of the parameters of the A department platform and the B department platform is as follows:
在信用分计算模型中,需要根据样本身份信息维度的所有信息计算出样本在该维度下的得分,在该***中,由于采用的计算模型权重迭代过程中只需要A部门平台的参数的梯度和B部门平台的参数的梯度,因此,首先A部门平台和B部门平台分别利用梯度计算公式进行训练,得到各自的参数的梯度和损失,然后将梯度和损失加密后发送给平台C,最后平台C中对A部门平台和B部门平台发送的梯度和损失进行解密,并基于A部门平台和B部门平台的损失计算融合损失,如果融合损失小于阈值,则终止训练;否则,下发梯度和损失到A部门平台和B部门平台,继续进行训练。In the credit score calculation model, it is necessary to calculate the score of the sample in this dimension based on all the information of the sample's identity information dimension. In this system, due to the calculation model weight iterative process, only the gradient and the parameters of the A department platform are needed. The gradient of the parameters of the B department platform. Therefore, first, the A department platform and the B department platform respectively use the gradient calculation formula to train to obtain the gradient and loss of their respective parameters. Then the gradient and loss are encrypted and sent to the platform C, and finally the platform C Decrypt the gradient and loss sent by the A department platform and the B department platform, and calculate the fusion loss based on the loss of the A department platform and the B department platform. If the fusion loss is less than the threshold, the training will be terminated; otherwise, the gradient and loss will be issued to The A department platform and the B department platform continue to train.
在这个过程中,A部门平台和B部门平台之间、平台C和A部门 平台之间、平台C和B部门平台之间相互传输的是加密后的梯度,由于各自信息的不可恢复,保证了各平台信息的安全性。In this process, the encrypted gradient is transmitted between the platform of department A and department B, between platform C and department A, and between platform C and department B. As the respective information is unrecoverable, it guarantees The security of information on each platform.
需要注意的是,梯度表示的是每个平台中模型的参数的导数,是对参数进行更新的中间变量。每个平台的损失是相应变量的函数。而梯度和损失可以利用现有的计算公式进行计算,本发明实施例不予赘述。It should be noted that the gradient represents the derivative of the parameters of the model in each platform, and is an intermediate variable for updating the parameters. The loss of each platform is a function of the corresponding variable. The gradient and loss can be calculated by using existing calculation formulas, which are not repeated in the embodiment of the present invention.
为了进一步阐述本发明实施例的技术思想,现结合具体的应用场景,对本发明实施例的技术方案进行说明。In order to further illustrate the technical ideas of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will now be described in combination with specific application scenarios.
以个人的信用分析以及评估信用分为例,本发明实施例的数据融合的方法可以实现对各政府部门和各互联网平台数据进行自动存入云平台,高效处理(包括:异常值处理、缺失值补充、离散化处理),定性定量分析和对多部门的数据进行安全快速的融合等功能;同时能够应用基于分布式并行计算的机器学习模型实时读取、分析海量信用数据,筛查异常数据,自动生成个人的信用数据分析报告,提交给用户,并将收集、处理、分析后的数据输入到信用评分云模型(即融合模型)中,输出信用分。Taking personal credit analysis and credit evaluation as an example, the data fusion method of the embodiment of the present invention can automatically store the data of various government departments and various Internet platforms in the cloud platform, and efficiently process (including: abnormal value processing, missing value Supplement, discretization processing), qualitative and quantitative analysis, and safe and fast fusion of data from multiple departments; at the same time, it can use the machine learning model based on distributed parallel computing to read and analyze massive credit data in real time, and screen for abnormal data. The personal credit data analysis report is automatically generated, submitted to the user, and the collected, processed, and analyzed data is input into the credit score cloud model (ie, the fusion model), and the credit score is output.
如图3所示,本发明一个可参考实施例的数据融合的方法的主要流程主要包括以下三个步骤;As shown in FIG. 3, the main process of a data fusion method that can refer to the embodiment of the present invention mainly includes the following three steps;
第一步:数据预处理The first step: data preprocessing
由于最终建模所需的数据可以来自于学校、银行或行政单位等各个部门和互联网平台,因此首先需要在各个云平台对各部门数据分别进行预处理。在此阶段,通过分别下发任务至数据处理方,使各个部门的数据预处理相互独立。预处理主要包括数据收集、数据探查、数据处理三个环节;Since the data required for final modeling can come from various departments such as schools, banks, or administrative units and Internet platforms, it is necessary to preprocess the data of each department separately on each cloud platform. At this stage, the data preprocessing of each department is independent of each other by separately issuing tasks to the data processing party. Preprocessing mainly includes three links: data collection, data exploration, and data processing;
1)在数据收集阶段,主要是将各个政府部门数据和互联网平台数据分别存入(同步传输)到各自的云平台。首先,各个部门确定各自需要同步到云平台的数据表,然后将这些表从本地生产环境同步到云平台。1) In the data collection stage, it is mainly to store (synchronously transmit) the data of various government departments and Internet platform data to their respective cloud platforms. First, each department determines the data tables that need to be synchronized to the cloud platform, and then synchronizes these tables from the local production environment to the cloud platform.
2)在数据探查阶段,主要是完成各个云平台每个原始数据表的数量和质量探查、数据表关联、数据字段维度划分。首先,分析确定每个云平台下多个原始数据表的逻辑关系(根据数据表主键确定逻辑关系),对每个原始数据表进行数据规模和数据质量探查,包括每个原始数据表的样本数量、字段数量、字段null值数量等;然后,从这些原始数据表中找出数据规模和数据质量满足要求的基础数据表,将这些基础数据表合并为一个关联数据表,此后各部门的数据处理都是对此关联数据表进行处理;最后为了提高***对个人信用的全面刻画能力,在各个平台完成数据表关联之后,将关联数据表的所有字段分成五个维度,即:身份特征、消费能力、信用历史、行为偏好、负债情况,给每个字段添加维度标签(注:一个字段可能有多个维度标签)。2) In the data exploration stage, it is mainly to complete the quantity and quality exploration of each original data table of each cloud platform, data table association, and data field dimension division. First, analyze and determine the logical relationship of multiple original data tables under each cloud platform (determine the logical relationship according to the primary key of the data table), and conduct data scale and data quality exploration for each original data table, including the sample number of each original data table , The number of fields, the number of field null values, etc.; then, from these original data tables, find out the basic data tables whose data size and data quality meet the requirements, merge these basic data tables into a linked data table, and then process the data of each department All of the related data tables are processed; in the end, in order to improve the system's ability to comprehensively describe personal credit, after each platform completes the data table association, all the fields of the related data table are divided into five dimensions, namely: identity characteristics, consumption ability , Credit history, behavior preference, debt situation, add dimension labels to each field (Note: a field may have multiple dimension labels).
3)在数据处理阶段,主要包括数据清洗、异常值的剔除、缺失值的补充,变量离散化等;首先是对各平台的关联数据表进行数据清洗,对于字段取值为空或null(空值)数量超过空字段阈值的样本进行剔除。然后是异常值的剔除和缺失值的补充,剔除的方法是采用箱线图结合字段的具体物理意义的方式进行去除;缺失值的补充,如果缺失数量较少,采用平均数、众数的方式填充,缺失过多的话,采用监督学习的方式进行预测填充,如随机森林等;最后是变量的离散化,主要是为了体现连续变量在不同的取值范围对模型影响不一样,将连续变量划份成离散变量,方法包括等频划份、等距划份或最优划份等。3) In the data processing stage, it mainly includes data cleaning, removal of outliers, supplementation of missing values, discretization of variables, etc.; first, data cleaning is performed on the associated data tables of each platform, and the value of the field is empty or null (empty Value) samples whose number exceeds the empty field threshold are eliminated. Then there is the elimination of outliers and the supplement of missing values. The method of elimination is to remove by combining the box plot with the specific physical meaning of the field; for the supplement of missing values, if the number of missing values is small, the average and mode are used. If there is too much filling or missing, use supervised learning methods for prediction filling, such as random forest, etc.; the last is the discretization of variables, mainly to reflect the different effects of continuous variables on the model in different value ranges. Division into discrete variables, methods include equal frequency division, equidistant division or optimal division, etc.
第二步:特征分析Step 2: Characteristic analysis
在各自云平台对各部门数据进行分别处理后,需要对各部门数据进行特征分析,包括:特征衍生、特征筛选;After the respective cloud platforms have processed the data of each department separately, it is necessary to analyze the characteristics of the data of each department, including: feature derivation and feature screening;
1)特征衍生是根据从各部门的各种数据类型中挖掘出影响个人信用的新变量(特征)。1) Feature derivation is based on mining new variables (features) that affect personal credit from various data types in various departments.
一种是采用无监督的特征衍生方法得到数据特征,数据特征主要包括以下几类:One is to use unsupervised feature derivation methods to obtain data features. Data features mainly include the following categories:
计数:过去12个月内申请贷款的总次数;Count: the total number of loan applications in the past 12 months;
求和:过去3个月网购金额;Sum: the amount of online purchases in the past 3 months;
比例:贷款人贷款金额与授信额度的占比;Ratio: the ratio of the loan amount of the lender to the credit line;
时间差:第一次开户距今时长;Time difference: the time since the first account opening;
波动率:过去3年工作时间的标准差;Volatility: the standard deviation of working hours in the past 3 years;
另一种是采用有监督的衍生方式,采用随机森林算法衍生得到高维非线性特征,提高信用评分云模型的非线性表达能力。The other is to use a supervised derivation method, using random forest algorithm to derive high-dimensional nonlinear features, and improve the nonlinear expression ability of the credit scoring cloud model.
2)特征筛选是从海量数据中分析现有的变量对个人信用的影响力,通过利用机器学习算法计算出各变量与个人信用的相关程度(其中相关程度可以通过计算出各个变量和标签之间的皮尔逊相关系数),此外,对于相关程度的检验,常见的标准包括卡方检验、信息增益、基尼指数和相关系数等。然后根据各个变量的相关程度、反映贷款人的还款能力和还款意愿筛选出待用特征,最终筛选出的五种维度的待用特征,具体见下表:2) Feature screening is to analyze the influence of existing variables on personal credit from massive data, and calculate the degree of correlation between each variable and personal credit by using machine learning algorithms (the degree of correlation can be calculated by calculating the relationship between each variable and label Pearson correlation coefficient). In addition, for the test of correlation degree, common standards include chi-square test, information gain, Gini index and correlation coefficient. Then according to the correlation degree of each variable, reflecting the lender's repayment ability and willingness to repay, the unused features are screened out, and the five dimensions of unused features are finally screened out, as shown in the following table:
第三步:数据融合Step 3: Data Fusion
完成各部门数据的数据处理和特征分析后,需要对来自各个部门的数据进行联合建模,实现对个人信用的全方位多角度刻画。After completing the data processing and characteristic analysis of the data of each department, it is necessary to jointly model the data from each department to achieve a comprehensive and multi-angle characterization of personal credit.
数据的融合是在训练模型抽取数据的过程中,采用联合加密技术方案,确保各部门信息的安全性,实现对个人信用的全方位多角度刻画;Data fusion is the use of joint encryption technology solutions in the process of extracting data from the training model to ensure the security of information in various departments and achieve a comprehensive and multi-angle characterization of personal credit;
首先,将各个平台的经过处理分析后的关联数据表根据待用特征进行数据对齐操作,分配统一标识(例如:身份证号、手机号)。First, perform data alignment operations on the associated data tables after processing and analysis of each platform according to the features to be used, and assign a unified identifier (for example: ID card number, mobile phone number).
然后,由于需要从五个维度分开建模,因此在每个维度建模过程中,根据各个平台关联表字段的维度标签和所对应的待用特征,选择每个维度建模时需要从各个平台抽取的字段,Then, because it needs to be modeled separately from five dimensions, in the modeling process of each dimension, according to the dimension label of each platform association table field and the corresponding to-be-used feature, you need to select each dimension to model from each platform Extracted fields,
最后,在各个维度建模过程中,开始从各个平台抽取相应字段,在具体的抽取过程中,从各个云平台抽取的是待用特征的梯度,基于待用特征的梯度建立融合模型。Finally, in the process of modeling each dimension, the corresponding fields are extracted from each platform. In the specific extraction process, the gradient of the features to be used is extracted from each cloud platform, and a fusion model is established based on the gradient of the features to be used.
所建立的融合模型可以计算某标准下的个人信用分值。在完成个人信用数据的收集、处理、分析后,信用评分模型(即融合模型)会根据用户的各个维度的信用信息(即融合信息)输出一个信用分数(即融合分值)。在融合信用信息后,可以为用户返回对应的信用报告,包括信用分数分析、影响信用分数变化的关键事件。The established fusion model can calculate the personal credit score under a certain standard. After completing the collection, processing, and analysis of personal credit data, the credit scoring model (ie, fusion model) will output a credit score (ie, fusion score) according to the user's credit information (ie, fusion information) in various dimensions. After integrating the credit information, the corresponding credit report can be returned to the user, including credit score analysis and key events affecting credit score changes.
在信用分数分析中,将会对用户的信用分数从时间维度上的变化进行分析、和从在所处地区、所处行业从空间维度上进行分析;在分析影响个人信用分数关键事件上,***平台通过基于分布式并行计算的机器学习模型实时读取、分析海量信用数据,得到用户的各个维度信用数据对信用分数的影响权重,并根据影响权重得到导致用户信用分数出现变化的关键事件。In the credit score analysis, the user’s credit score will be analyzed from the time dimension, and from the region and industry from the spatial dimension; in the analysis of key events affecting personal credit score, the system will The platform reads and analyzes massive credit data in real time through a machine learning model based on distributed parallel computing, and obtains the weight of the user's various dimensions of credit data's influence on the credit score, and obtains the key events that cause the user's credit score to change according to the influence weight.
此外,建模过程中涉及到多机构数据的场景均可运用本发明实施例的数据融合的方法,例如:营业厅选址(运营商数据、网站数据)、客户推荐(银行或网站)等,本发明实施例不再一一列举。In addition, scenarios involving multi-institutional data in the modeling process can use the data fusion method of the embodiment of the present invention, such as: business hall location (operator data, website data), customer recommendation (bank or website), etc. The embodiments of the present invention will not be listed one by one.
图4是根据本发明实施例的数据融合的装置的主要模块的示意图。Fig. 4 is a schematic diagram of main modules of a data fusion device according to an embodiment of the present invention.
如图4所示,本发明实施例的数据融合的装置400包括:合并模块401、分析模块402、抽取模块403和建模模块404。As shown in FIG. 4, the data fusion device 400 of the embodiment of the present invention includes: a merging module 401, an analysis module 402, an extraction module 403, and a modeling module 404.
其中,among them,
合并模块401,用于对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为所述关联数据表中的字段添加维度标签;The merging module 401 is configured to filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table;
分析模块402,用于对所述关联数据表中的字段进行特征分析,选取待用特征;The analysis module 402 is configured to perform feature analysis on the fields in the associated data table and select features to be used;
抽取模块403,用于根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度;An extraction module 403, configured to extract gradients from each of the associated data tables according to the dimension label and the feature to be used;
建模模块404,用于基于所述梯度建立融合模型。The modeling module 404 is configured to establish a fusion model based on the gradient.
在本发明实施例中,所述合并模块401还用于:In the embodiment of the present invention, the merging module 401 is further configured to:
从各个数据源收集原始数据表;Collect raw data tables from various data sources;
分析所述原始数据表的样本数量、字段数量和空值数量;Analyze the number of samples, the number of fields, and the number of null values of the original data table;
基于所述样本数量、所述字段数量和所述空值数量从所述原始数据表筛选出基础数据表;Filtering out a basic data table from the original data table based on the number of samples, the number of fields, and the number of null values;
基于所述基础数据表的主键将同一数据源的所述基础数据表合并为关联数据表。Combining the basic data tables of the same data source into an associated data table based on the primary key of the basic data table.
在本发明实施例中,所述合并模块401进一步用于:In the embodiment of the present invention, the merging module 401 is further configured to:
剔除所述关联数据表中含有空值字段或空值字段的数量超过空字段阈值的样本;Removing samples in the associated data table that contain null value fields or the number of null value fields exceeds the null field threshold;
利用箱线图结合字段意义的方式去除异常字段;Use box plots and field meanings to remove abnormal fields;
采用平均数、众数或预测值补充空值字段;以及Use average, mode, or predicted values to supplement empty value fields; and
对所述关联数据表进行离散化处理。Discretize the associated data table.
在本发明实施例中,所述分析模块402还用于:In the embodiment of the present invention, the analysis module 402 is further configured to:
对所述关联数据表中的字段进行特征衍生,得到数据特征和高维非线性特征;Perform feature derivation on the fields in the associated data table to obtain data features and high-dimensional nonlinear features;
分别计算所述数据特征和所述高维非线性特征的皮尔逊系数;Respectively calculating the Pearson coefficients of the data feature and the high-dimensional nonlinear feature;
基于所述皮尔逊系数从所述数据特征和所述高维非线性特征中选取待用特征。Select a feature to be used from the data feature and the high-dimensional nonlinear feature based on the Pearson coefficient.
在本发明实施例中,所述抽取模块403还用于:In the embodiment of the present invention, the extraction module 403 is also used to:
根据所述待用特征对各个数据源对应的所述关联数据表进行对齐操作,并为目标分配统一标识;Aligning the associated data table corresponding to each data source according to the features to be used, and assigning a unified identifier to the target;
根据所述维度标签和所述待用特征从各个所述关联数据表中抽取数据源的参数的梯度和所述关联数据表的损失,并基于各个所述关联数据表的损失计算融合损失;Extract the gradient of the parameter of the data source and the loss of the associated data table from each of the associated data tables according to the dimensional label and the features to be used, and calculate a fusion loss based on the loss of each of the associated data tables;
若所述融合损失小于损失阈值,则终止抽取;If the fusion loss is less than the loss threshold, the extraction is terminated;
若所述融合损失大于或等于损失阈值,则将更新各个数据源的参数,并重新抽取数据源的参数的梯度和所述关联数据表的损失。If the fusion loss is greater than or equal to the loss threshold, the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
此外,所述数据融合的装置400还包括计算模块(图中并未示出),用于:In addition, the data fusion device 400 further includes a calculation module (not shown in the figure) for:
从各个所述关联数据表中提取目标的待用特征信息,基于所述待用特征信息生成融合信息;Extracting target feature information for use from each of the associated data tables, and generating fusion information based on the feature information for use;
将所述融合信息输入所述融合模型,以根据所述待用特征和所述梯度对所述融合信息进行计算,得到目标的融合分值;以及Inputting the fusion information into the fusion model to calculate the fusion information according to the features to be used and the gradient to obtain the fusion score of the target; and
在目标的融合分值变化时,对所述待用特征信息进行迭代计算,得到所述待用特征对目标的影响权重。When the fusion score of the target changes, iterative calculation is performed on the to-be-used feature information to obtain the influence weight of the to-be-used feature on the target.
根据本发明实施例的数据融合的装置可以看出,因为采用对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为关联数据表中的字段添加维度标签;对关联数据表中的字段进行特征分析,选取待用特征;根据维度标签和待用特征从各个关联数据表中抽取梯度;基于梯度建立融合模型的技术手段,通过融合模型构建统一的、安全的融合信息评分体系,所以克服了缺少统一的、安全的数据融合方法;在对来自异构数据源的数据进行融合过程中,存在信息泄露的安全问题;以及无法为构建统一的融合信息评分体系提供相同标准和格式的数据的技术问题,进而达到对不同数据源的数据进行收集和高效处理,对不同结构的数据进行安全快速的融合,为构建 统一的融合信息评分体系提供相同标准和格式的数据的技术效果。According to the data fusion device of the embodiment of the present invention, it can be seen that the original data table of each data source is filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields in the associated data table and select the features to be used; extract gradients from each associated data table according to the dimension label and the features to be used; build a fusion model based on the gradient, and build a unified and safe The fusion information scoring system overcomes the lack of a unified and safe data fusion method; in the process of fusing data from heterogeneous data sources, there is a security problem of information leakage; and it cannot provide for the construction of a unified fusion information scoring system The technical problems of data of the same standard and format, and then achieve the collection and efficient processing of data from different data sources, the safe and rapid fusion of data of different structures, and provide data of the same standard and format for building a unified integrated information scoring system Technical effect.
图5示出了可以应用本发明实施例的数据融合的方法或数据融合的装置的示例性***架构500。FIG. 5 shows an exemplary system architecture 500 of the data fusion method or data fusion device to which the embodiment of the present invention can be applied.
如图5所示,***架构500可以包括终端设备501、502、503,网络504和服务器505。网络504用以在终端设备501、502、503和服务器505之间提供通信链路的介质。网络504可以包括各种连接类型,例如有线、无线通信链路或者光纤电缆等等。As shown in FIG. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.
用户可以使用终端设备501、502、503通过网络504与服务器505交互,以接收或发送消息等。终端设备501、502、503上可以安装有各种通讯客户端应用,例如购物类应用、网页浏览器应用、搜索类应用、即时通信工具、邮箱客户端、社交平台软件等。The user can use the terminal devices 501, 502, 503 to interact with the server 505 via the network 504 to receive or send messages and so on. Various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social platform software, may be installed on the terminal devices 501, 502, and 503.
终端设备501、502、503可以是具有显示屏并且支持网页浏览的各种电子设备,包括但不限于智能手机、平板电脑、膝上型便携计算机和台式计算机等等。The terminal devices 501, 502, and 503 may be various electronic devices that have a display screen and support web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and so on.
服务器505可以是提供各种服务的服务器,例如对用户利用终端设备501、502、503所浏览的购物类网站提供支持的后台管理服务器。后台管理服务器可以对接收到的产品信息查询请求等数据进行分析等处理,并将处理结果(例如目标推送信息、产品信息)反馈给终端设备。The server 505 may be a server that provides various services, for example, a background management server that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The background management server can analyze and process the received product information query request and other data, and feed back the processing result (for example, target push information, product information) to the terminal device.
需要说明的是,本发明实施例所提供的数据融合的方法一般由服务器505执行,相应地,数据融合的装置一般设置于服务器505中。It should be noted that the data fusion method provided in the embodiment of the present invention is generally executed by the server 505, and accordingly, the data fusion device is generally set in the server 505.
应该理解,图5中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要,可以具有任意数目的终端设备、网络和服务器。It should be understood that the numbers of terminal devices, networks, and servers in FIG. 5 are merely illustrative. According to implementation needs, there can be any number of terminal devices, networks and servers.
下面参考图6,其示出了适于用来实现本发明实施例的终端设备的计算机***600的结构示意图。图6示出的终端设备仅仅是一个示例,不应对本发明实施例的功能和使用范围带来任何限制。Reference is now made to FIG. 6, which shows a schematic structural diagram of a computer system 600 suitable for implementing the terminal device of the embodiment of the present invention. The terminal device shown in FIG. 6 is only an example, and should not bring any limitation to the function and application scope of the embodiment of the present invention.
如图6所示,计算机***600包括中央处理单元(CPU)601,其可以根据存储在只读存储器(ROM)602中的程序或者从存储部分608加载到随机访问存储器(RAM)603中的程序而执行各种适当的动作和处理。在RAM 603中,还存储有***600操作所需的各种程序和数据。CPU 601、ROM 602以及RAM 603通过总线604彼此相连。输入/输出(I/O)接口605也连接至总线604。As shown in FIG. 6, the computer system 600 includes a central processing unit (CPU) 601, which can be based on a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603 And perform various appropriate actions and processing. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to the bus 604.
以下部件连接至I/O接口605:包括键盘、鼠标等的输入部分606;包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分607;包括硬盘等的存储部分608;以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分609。通信部分609经由诸如因特网的网络执行通信处理。驱动器610也根据需要连接至I/O接口605。可拆卸介质611,诸如磁盘、光盘、磁光盘、半导体存储器等等,根据需要安装在驱动器610上,以便于从其上读出的计算机程序根据需要被安装入存储部分608。The following components are connected to the I/O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and speakers, etc.; a storage part 608 including a hard disk, etc. ; And a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that the computer program read from it is installed into the storage part 608 as needed.
特别地,根据本发明公开的实施例,上文参考流程图描述的过程可以被实现为计算机软件程序。例如,本发明公开的实施例包括一种计算机程序产品,其包括承载在计算机可读介质上的计算机程序,该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中,该计算机程序可以通过通信部分609从网络上被下载和安装,和/或从可拆卸介质611被安装。在该计算机程序被中央处理单元(CPU)601执行时,执行本发明的***中限定的上述功能。In particular, according to the disclosed embodiments of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the disclosed embodiments of the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication part 609, and/or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.
需要说明的是,本发明所示的计算机可读介质可以是计算机可读 信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装置或器件,或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于:具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本发明中,计算机可读存储介质可以是任何包含或存储程序的有形介质,该程序可以被指令执行***、装置或者器件使用或者与其结合使用。而在本发明中,计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号,其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式,包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质,该计算机可读介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输,包括但不限于:无线、电线、光缆、RF等等,或者上述的任意合适的组合。It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
附图中的流程图和框图,图示了按照本发明各种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分,上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意,在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个接连地表示的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图或流程图中的每个方框、以及框图或流程图中的方框的组合,可以用执行规定的功能或操作的专用的基于硬件的***来实现,或者可以用专用硬件与计算机指令的组合来实现。The flowcharts and block diagrams in the drawings illustrate the possible implementation of the system architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the above-mentioned module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be It is realized by a combination of dedicated hardware and computer instructions.
描述于本发明实施例中所涉及到的模块可以通过软件的方式实现,也可以通过硬件的方式来实现。所描述的模块也可以设置在处理器中,例如,可以描述为:一种处理器包括合并模块、分析模块、抽取模块和建模模块。其中,这些模块的名称在某种情况下并不构成对该模块本身的限定,例如,建模模块还可以被描述为“基于所述梯度建立融合模型的模块”。The modules involved in the described embodiments of the present invention can be implemented in software or hardware. The described modules can also be provided in the processor, for example, it can be described as: a processor includes a merging module, an analysis module, an extraction module, and a modeling module. Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances. For example, the modeling module can also be described as a “module that builds a fusion model based on the gradient”.
作为另一方面,本发明还提供了一种计算机可读介质,该计算机可读介质可以是上述实施例中描述的设备中所包含的;也可以是单独存在,而未装配入该设备中。上述计算机可读介质承载有一个或者多个程序,当上述一个或者多个程序被一个该设备执行时,使得该设备包括:步骤S101:对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为关联数据表中的字段添加维度标签;步骤S102:对关联数据表中的字段进行特征分析,选取待用特征;步骤S103:根据维度标签和待用特征从各个关联数据表中抽取梯度;步骤S104:基于梯度建立融合模型。As another aspect, the present invention also provides a computer-readable medium. The computer-readable medium may be included in the device described in the foregoing embodiment; or it may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the device includes: Step S101: filtering and merging the original data tables of each data source to obtain each The associated data table of the data source, and add dimension labels to the fields in the associated data table; Step S102: Perform feature analysis on the fields in the associated data table, and select the features to be used; Step S103: According to the dimension labels and features to be used The gradient is extracted from the associated data table; step S104: a fusion model is established based on the gradient.
根据本发明实施例的技术方案,因为采用对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为关联数据表中的字段添加维度标签;对关联数据表中的字段进行特征分析,选取待用特征;根据维度标签和待用特征从各个关联数据表中抽取梯度;基于梯度建立融合模型的技术手段,通过融合模型构建统一的、安全的融合信息评分体系,所以克服了缺少统一的、安全的数据融合方法;在对来自异构数据源的数据进行融合过程中,存在信息泄露的安全问题;以及无法为构建统一的融合信息评分体系提供相同标准和格式的数据的技术问题,进而达到对不同数据源的数据进行收集和高效处理,对不同结构的数据进行安全快速的融合,为构建统一的融合信息评分体系提供相同标准和格式的数据的技术效果。According to the technical solution of the embodiment of the present invention, the original data table of each data source is filtered and merged to obtain the associated data table of each data source, and dimension labels are added to the fields in the associated data table; Perform feature analysis on the fields of, and select the features to be used; extract gradients from each associated data table according to the dimension labels and features to be used; technical means to establish a fusion model based on the gradient, and build a unified and secure fusion information scoring system through the fusion model. Therefore, it overcomes the lack of a unified and secure data fusion method; in the process of fusing data from heterogeneous data sources, there are security problems of information leakage; and the failure to provide the same standards and formats for building a unified fusion information scoring system The technical problems of data, and then achieve the collection and efficient processing of data from different data sources, the safe and rapid integration of data of different structures, and the technical effect of providing data of the same standard and format for building a unified fusion information scoring system.
上述具体实施方式,并不构成对本发明保护范围的限制。本领域技术人员应该明白的是,取决于设计要求和其他因素,可以发生各种各样的修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等,均应包含在本发明保护范围之内。The foregoing specific implementations do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that, depending on design requirements and other factors, various modifications, combinations, sub-combinations, and substitutions can occur. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
- 一种数据融合的方法,其特征在于,包括:A method of data fusion, characterized in that it includes:对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为所述关联数据表中的字段添加维度标签;Filtering and merging the original data tables of each data source to obtain the associated data table of each data source, and adding dimension labels to the fields in the associated data table;对所述关联数据表中的字段进行特征分析,选取待用特征;Perform feature analysis on the fields in the associated data table, and select features to be used;根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度;Extract a gradient from each of the associated data tables according to the dimension label and the feature to be used;基于所述梯度建立融合模型。A fusion model is established based on the gradient.
- 根据权利要求1所述的方法,其特征在于,对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表包括:The method according to claim 1, wherein the filtering and merging the original data tables of each data source to obtain the associated data table of each data source comprises:从各个数据源收集原始数据表;Collect raw data tables from various data sources;分析所述原始数据表的样本数量、字段数量和空值数量;Analyze the number of samples, the number of fields, and the number of null values of the original data table;基于所述样本数量、所述字段数量和所述空值数量从所述原始数据表筛选出基础数据表;Filtering out a basic data table from the original data table based on the number of samples, the number of fields, and the number of null values;基于所述基础数据表的主键将同一数据源的所述基础数据表合并为关联数据表。Combining the basic data tables of the same data source into an associated data table based on the primary key of the basic data table.
- 根据权利要求1所述的方法,其特征在于,在对所述关联数据表中的字段进行特征分析,选取待用特征之前还包括:The method according to claim 1, characterized in that, before performing feature analysis on the fields in the associated data table, the method further comprises:剔除所述关联数据表中含有空值字段或空值字段的数量超过空字段阈值的样本;Removing samples in the associated data table that contain null value fields or the number of null value fields exceeds the null field threshold;利用箱线图结合字段意义的方式去除异常字段;Use box plots and field meanings to remove abnormal fields;采用平均数、众数或预测值补充空值字段;以及Use average, mode, or predicted values to supplement empty value fields; and对所述关联数据表进行离散化处理。Discretize the associated data table.
- 根据权利要求1所述的方法,其特征在于,对所述关联数据表中的字段进行特征分析,选取待用特征包括:The method according to claim 1, wherein performing feature analysis on the fields in the associated data table, and selecting features to be used comprises:对所述关联数据表中的字段进行特征衍生,得到数据特征和高维 非线性特征;Perform feature derivation on the fields in the associated data table to obtain data features and high-dimensional nonlinear features;分别计算所述数据特征和所述高维非线性特征的皮尔逊系数;Respectively calculating the Pearson coefficients of the data feature and the high-dimensional nonlinear feature;基于所述皮尔逊系数从所述数据特征和所述高维非线性特征中选取待用特征。Select a feature to be used from the data feature and the high-dimensional nonlinear feature based on the Pearson coefficient.
- 根据权利要求1所述的方法,其特征在于,根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度包括:The method of claim 1, wherein extracting gradients from each of the associated data tables according to the dimension label and the feature to be used comprises:根据所述待用特征对各个数据源对应的所述关联数据表进行对齐操作,并为目标分配统一标识;Aligning the associated data table corresponding to each data source according to the features to be used, and assigning a unified identifier to the target;根据所述维度标签和所述待用特征从各个所述关联数据表中抽取数据源的参数的梯度和所述关联数据表的损失,并基于各个所述关联数据表的损失计算融合损失;Extract the gradient of the parameter of the data source and the loss of the associated data table from each of the associated data tables according to the dimensional label and the features to be used, and calculate a fusion loss based on the loss of each of the associated data tables;若所述融合损失小于损失阈值,则终止抽取;If the fusion loss is less than the loss threshold, the extraction is terminated;若所述融合损失大于或等于损失阈值,则将更新各个数据源的参数,并重新抽取数据源的参数的梯度和所述关联数据表的损失。If the fusion loss is greater than or equal to the loss threshold, the parameters of each data source will be updated, and the gradient of the parameters of the data source and the loss of the associated data table will be extracted again.
- 根据权利要求5所述的方法,其特征在于,所述方法还包括:The method of claim 5, wherein the method further comprises:从各个所述关联数据表中提取目标的待用特征信息,基于所述待用特征信息生成融合信息;Extracting target feature information for use from each of the associated data tables, and generating fusion information based on the feature information for use;将所述融合信息输入所述融合模型,以根据所述待用特征和所述梯度对所述融合信息进行计算,得到目标的融合分值;以及Inputting the fusion information into the fusion model to calculate the fusion information according to the features to be used and the gradient to obtain the fusion score of the target; and在目标的融合分值变化时,对所述待用特征信息进行迭代计算,得到所述待用特征对目标的影响权重。When the fusion score of the target changes, iterative calculation is performed on the to-be-used feature information to obtain the influence weight of the to-be-used feature on the target.
- 一种数据融合的装置,其特征在于,包括:A data fusion device, characterized in that it comprises:合并模块,用于对各个数据源的原始数据表进行筛选合并,得到每个数据源的关联数据表,并为所述关联数据表中的字段添加维度标签;The merging module is used to filter and merge the original data tables of each data source to obtain the associated data table of each data source, and add dimension labels to the fields in the associated data table;分析模块,用于对所述关联数据表中的字段进行特征分析,选取待用特征;The analysis module is used to perform feature analysis on the fields in the associated data table and select features to be used;抽取模块,用于根据所述维度标签和所述待用特征从各个所述关联数据表中抽取梯度;An extraction module, configured to extract gradients from each of the associated data tables according to the dimension label and the feature to be used;建模模块,用于基于所述梯度建立融合模型。The modeling module is used to establish a fusion model based on the gradient.
- 根据权利要求7所述的装置,其特征在于,所述装置还包括计算模块,用于:8. The device according to claim 7, wherein the device further comprises a calculation module for:从各个所述关联数据表中提取目标的待用特征信息,基于所述待用特征信息生成融合信息;Extracting target feature information for use from each of the associated data tables, and generating fusion information based on the feature information for use;将所述融合信息输入所述融合模型,以根据所述待用特征和所述梯度对所述融合信息进行计算,得到目标的融合分值;以及Inputting the fusion information into the fusion model to calculate the fusion information according to the features to be used and the gradient to obtain the fusion score of the target; and在目标的融合分值变化时,对所述待用特征信息进行迭代计算,得到所述待用特征对目标的影响权重。When the fusion score of the target changes, iterative calculation is performed on the to-be-used feature information to obtain the influence weight of the to-be-used feature on the target.
- 一种数据融合的电子设备,其特征在于,包括:A data fusion electronic device, characterized in that it includes:一个或多个处理器;One or more processors;存储装置,用于存储一个或多个程序,Storage device for storing one or more programs,当所述一个或多个程序被所述一个或多个处理器执行,使得所述一个或多个处理器实现如权利要求1-6中任一所述的方法。When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.
- 一种计算机可读介质,其上存储有计算机程序,其特征在于,所述程序被处理器执行时实现如权利要求1-6中任一所述的方法。A computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method according to any one of claims 1-6.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910362476.2 | 2019-04-30 | ||
CN201910362476.2A CN110119413B (en) | 2019-04-30 | 2019-04-30 | Data fusion method and device |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020220810A1 true WO2020220810A1 (en) | 2020-11-05 |
Family
ID=67520291
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/076895 WO2020220810A1 (en) | 2019-04-30 | 2020-02-27 | Data fusion method and apparatus |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110119413B (en) |
WO (1) | WO2020220810A1 (en) |
Families Citing this family (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110119413B (en) * | 2019-04-30 | 2024-06-18 | 京东城市(南京)科技有限公司 | Data fusion method and device |
CN110544023A (en) * | 2019-08-16 | 2019-12-06 | 北京慧辰资道资讯股份有限公司 | Enterprise regional contribution data evaluation system and evaluation method thereof |
CN110738523B (en) * | 2019-10-15 | 2023-03-24 | 北京经纬恒润科技股份有限公司 | Maintenance order quantity prediction method and device |
CN110874369A (en) * | 2019-10-25 | 2020-03-10 | 广州纳斯威尔信息技术有限公司 | Multidimensional data fusion investigation system and method thereof |
CN110990450A (en) * | 2019-11-08 | 2020-04-10 | 中国检验检疫科学研究院 | Method and device for processing potential invasive livestock and poultry epidemic situation data |
CN110955659B (en) * | 2019-11-28 | 2023-02-21 | 第四范式(北京)技术有限公司 | Method and system for processing data table |
CN111191688A (en) * | 2019-12-16 | 2020-05-22 | 北京淇瑀信息科技有限公司 | User staging number management method and device and electronic equipment |
CN111143372B (en) * | 2019-12-30 | 2023-09-26 | 网易(杭州)网络有限公司 | Data processing method and device |
CN111475532B (en) * | 2020-03-05 | 2023-11-03 | 拉扎斯网络科技(上海)有限公司 | Data processing optimization method and device, storage medium and terminal |
CN111242520B (en) * | 2020-04-24 | 2021-03-02 | 北京淇瑀信息科技有限公司 | Feature synthesis model generation method and device and electronic equipment |
CN111242519B (en) * | 2020-04-24 | 2020-07-17 | 北京淇瑀信息科技有限公司 | User characteristic data generation method and device and electronic equipment |
CN111625545B (en) * | 2020-06-08 | 2023-09-01 | 北京创鑫旅程网络技术有限公司 | Data fusion method, data fusion device and storage medium |
CN111753331B (en) * | 2020-06-28 | 2023-09-19 | 中国银行股份有限公司 | Data bleaching method and device |
CN112000841B (en) * | 2020-07-29 | 2023-09-26 | 北京达佳互联信息技术有限公司 | Electronic tag data processing method and device, electronic equipment and storage medium |
CN112051992A (en) * | 2020-09-02 | 2020-12-08 | 天元大数据信用管理有限公司 | Data processing method and system based on credit investigation report generation |
CN111985194A (en) * | 2020-09-03 | 2020-11-24 | 深圳壹账通智能科技有限公司 | Data storage method and device, electronic equipment and storage medium |
CN114443639A (en) * | 2020-11-02 | 2022-05-06 | 第四范式(北京)技术有限公司 | Method and system for processing data table and automatically training machine learning model |
CN112380215B (en) * | 2020-11-17 | 2023-07-28 | 北京融七牛信息技术有限公司 | Automatic feature generation method based on cross aggregation |
CN112488561B (en) * | 2020-12-10 | 2024-05-14 | 广州市来利洪饼业有限公司 | Moon cake safety quality guaranteeing method and system based on short-time hot baking and ultraviolet rays |
CN112613977A (en) * | 2020-12-17 | 2021-04-06 | 中国建设银行股份有限公司 | Personal credit loan admission credit granting method and system based on government affair data |
CN113379945A (en) * | 2021-07-26 | 2021-09-10 | 陕西天行健车联网信息技术有限公司 | Vehicle driving behavior analysis device, method and system |
CN113535707B (en) * | 2021-08-05 | 2022-04-15 | 南京华飞数据技术有限公司 | Method for managing personnel information data based on big data |
CN114117052A (en) * | 2021-11-30 | 2022-03-01 | 深圳前海微众银行股份有限公司 | Method and device for classifying business data reports |
CN114169269B (en) * | 2021-12-08 | 2024-05-31 | 南京金崎新能源动力研究院有限公司 | Virtual driving simulation method for LED |
CN115185939B (en) * | 2022-09-07 | 2022-11-18 | 中航信移动科技有限公司 | Data processing method of multi-source data |
CN115983636B (en) * | 2022-12-26 | 2023-11-17 | 深圳市中政汇智管理咨询有限公司 | Risk assessment method, apparatus, device and storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106920147A (en) * | 2017-02-28 | 2017-07-04 | 华中科技大学 | A kind of commodity intelligent recommendation method that word-based vector data drives |
CN107526786A (en) * | 2017-08-01 | 2017-12-29 | 江苏速度信息科技股份有限公司 | The method and system that place name address date based on multi-source data is integrated |
CN108154430A (en) * | 2017-12-28 | 2018-06-12 | 上海氪信信息技术有限公司 | A kind of credit scoring construction method based on machine learning and big data technology |
CN110119413A (en) * | 2019-04-30 | 2019-08-13 | 京东城市(南京)科技有限公司 | The method and apparatus of data fusion |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109117864B (en) * | 2018-07-13 | 2020-02-28 | 华南理工大学 | Coronary heart disease risk prediction method, model and system based on heterogeneous feature fusion |
-
2019
- 2019-04-30 CN CN201910362476.2A patent/CN110119413B/en active Active
-
2020
- 2020-02-27 WO PCT/CN2020/076895 patent/WO2020220810A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106920147A (en) * | 2017-02-28 | 2017-07-04 | 华中科技大学 | A kind of commodity intelligent recommendation method that word-based vector data drives |
CN107526786A (en) * | 2017-08-01 | 2017-12-29 | 江苏速度信息科技股份有限公司 | The method and system that place name address date based on multi-source data is integrated |
CN108154430A (en) * | 2017-12-28 | 2018-06-12 | 上海氪信信息技术有限公司 | A kind of credit scoring construction method based on machine learning and big data technology |
CN110119413A (en) * | 2019-04-30 | 2019-08-13 | 京东城市(南京)科技有限公司 | The method and apparatus of data fusion |
Also Published As
Publication number | Publication date |
---|---|
CN110119413B (en) | 2024-06-18 |
CN110119413A (en) | 2019-08-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020220810A1 (en) | Data fusion method and apparatus | |
US20210200759A1 (en) | Systems and Methods for Data Mining of Historic Electronic Communication Exchanges to Identify Relationships, Patterns, and Correlations to Deal Outcomes | |
CN111427971B (en) | Business modeling method, device, system and medium for computer system | |
CN111046192A (en) | Identification method and device for bank case-involved account | |
US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
CN111949643A (en) | Data processing method and system based on business modeling | |
CN113836131A (en) | Big data cleaning method and device, computer equipment and storage medium | |
CN111563267A (en) | Method and device for processing federal characteristic engineering data | |
CN112417060A (en) | Method, device, equipment and computer readable medium for identifying enterprise relationship | |
CN110895761B (en) | After-sales service application information processing method and device | |
CN112990311A (en) | Method and device for identifying admitted client | |
CN112950359A (en) | User identification method and device | |
CN113761350A (en) | Data recommendation method, related device and data recommendation system | |
WO2023175413A1 (en) | Mutual exclusion data class analysis in data governance | |
US20230289325A1 (en) | Multi-Computer System for Controlling Data Relation and Redundancy | |
US11593740B1 (en) | Computing system for automated evaluation of process workflows | |
CN114565470A (en) | Financial product recommendation method based on artificial intelligence and related equipment thereof | |
CN111144694B (en) | Operation flow management method and device based on contact analysis | |
CN112487262A (en) | Data processing method and device | |
EP3555839A1 (en) | Multi-corporation venture plan validation and inclusion of time series geospatial markers in analyses employing an advanced decision platform | |
Bratosin | Boarding to Big data | |
CN117436594B (en) | Intelligent information management method and system for enterprise clients | |
Abdulkadium et al. | Raid Abd Alreda Shekan | |
CN116842052A (en) | Data processing method, device, equipment and medium based on integration model | |
CN113902032A (en) | Service data processing method and device, computer equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20798834 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20798834 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20798834 Country of ref document: EP Kind code of ref document: A1 |