CN116595589B - Secret sharing mechanism-based distributed support vector machine training method and system - Google Patents

Secret sharing mechanism-based distributed support vector machine training method and system Download PDF

Info

Publication number
CN116595589B
CN116595589B CN202310869601.5A CN202310869601A CN116595589B CN 116595589 B CN116595589 B CN 116595589B CN 202310869601 A CN202310869601 A CN 202310869601A CN 116595589 B CN116595589 B CN 116595589B
Authority
CN
China
Prior art keywords
participants
participant
training
support vector
secret
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310869601.5A
Other languages
Chinese (zh)
Other versions
CN116595589A (en
Inventor
张纪峰
王继民
赵延龙
郭金
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology Beijing USTB
Academy of Mathematics and Systems Science of CAS
Original Assignee
University of Science and Technology Beijing USTB
Academy of Mathematics and Systems Science of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology Beijing USTB, Academy of Mathematics and Systems Science of CAS filed Critical University of Science and Technology Beijing USTB
Priority to CN202310869601.5A priority Critical patent/CN116595589B/en
Publication of CN116595589A publication Critical patent/CN116595589A/en
Application granted granted Critical
Publication of CN116595589B publication Critical patent/CN116595589B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/08—Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords
    • H04L9/0816—Key establishment, i.e. cryptographic processes or cryptographic protocols whereby a shared secret becomes available to two or more parties, for subsequent use
    • H04L9/085—Secret sharing or secret splitting, e.g. threshold schemes
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Computer Security & Cryptography (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computer Hardware Design (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Complex Calculations (AREA)

Abstract

The invention relates to the technical field of support vector machines, in particular to a distributed support vector machine training method and system based on a secret sharing mechanism, wherein the method comprises the following steps: s1, decomposing an optimization problem of a Support Vector Machine (SVM) into N sub-optimization problems by using an Alternating Direction Multiplication Method (ADMM) algorithm; s2, respectively carrying out local solution on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics; s3, training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters. By adopting the method and the device, the targets of cooperative training and privacy protection of the completely distributed scene can be realized.

Description

Secret sharing mechanism-based distributed support vector machine training method and system
Technical Field
The invention relates to the technical field of support vector machines, in particular to a distributed support vector machine training method and system based on a secret sharing mechanism.
Background
The Support Vector Machine (SVM) is a widely used supervised learning (supervised learning) method, and is mainly used for constructing a generalized linear classifier for binary classification of data, and has been popular in text classification, handwriting character recognition and bioinformatics at present. In a real scene, the problem of huge training sample set exists, the hardware of a single machine can not support the requirement of directly operating all data, the accuracy of the classification result of an algorithm is difficult to ensure, and in addition, how to ensure the privacy safety in the implementation process of a support vector machine is also a problem to be solved urgently.
Disclosure of Invention
The embodiment of the invention provides a distributed support vector machine training method and system based on a secret sharing mechanism. The technical scheme is as follows:
in one aspect, a method for training a distributed support vector machine based on a secret sharing mechanism is provided, including:
s1, decomposing an optimization problem of a Support Vector Machine (SVM) into N sub-optimization problems by using an Alternating Direction Multiplication Method (ADMM) algorithm;
s2, respectively carrying out local solution on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics;
S3, training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters.
Optionally, the feature space of the input samples and labels is given in the SVM classification problemThe solved hyperplane is denoted +.>W is the normal vector of the hyperplane, b is the intercept, and the model of the SVM is:
（1）
where N is the number of samples，x i Represents the i-th sample, y i Represents the label corresponding to the ith sample, ζ i Is a relaxation variable, each sample corresponds to one relaxation variable, but the relaxation variables of points without outliers are all 0, which indicates the degree that the samples do not meet the constraint, fault tolerance is introduced, C is a penalty factor, and w and b are learned and output through a training method;
the Lagrangian function L of the solution model is shown in equation (2):
（2）
wherein alpha and mu are Lagrangian multipliers, alpha i ≥ 0， µ i ≥ 0；
The problem of the support vector machine is converted to solve the minimum and maximum problem of L by equation (2), with the goal of solving for the values of w, b and ζ that make L minimum by solving for a that makes L function maximum,the Lagrangian duality is applied here, the optimal solution of the original problem is solved by solving the dual problem, the solving difficulty can be reduced, the dual problem of the original problem is a maximum and minimum problem, w, b and ζ which enable the L function to be minimum are solved first, and then the maximum of alpha is solved for the L function, wherein the dual problem is shown as a formula (3):
（3）
Wherein α= (α) 1 , α 2 , . . . , α N ) Is a Lagrangian multiplier vector;
in the model training stage, the optimal problem is solved by combining a steepest descent method with an outlier penalty method, and the method comprises the following steps:
(1) External circulation
step 1: selecting an initial point alpha 0 Initial penalty factor M 1 =1, precision ε 1 > 0 , k := 1
step 2: to be used forFor an initial point, solving a constraint optimization problem as in equation (4):
（4）
solving F (alpha, M) by entering an inner circulation steepest descent method k ) Obtaining the minimum pointIs marked as->, wherein ：
（5）
step 3: if it isStopping the calculation to obtain the approximate minimum point +.>Otherwise let M k+1 =cM k Let k =k+1, turn step 1;
(2) Internal circulation
step 1: selection ofIs the initial point alpha of internal circulation 0 Precision epsilon 2 >0, let t: =1
step 2 determining the search direction for each alpha i And (3) calculating:
（6）
wherein ：
（7）
finally obtaining the search direction d t ：
（8）
step 3: from the slaveStarting from the edge d t Step length lambda of direction finding t
step 4: calculation of
step 5: if it isThe iteration is stopped, if not let t: =t+1, return step 2.
Alternatively, assuming n=2, the S1 specifically includes: the optimization problem of the SVM is decomposed into two sub-optimization problems f (x) and g (z):
（9）
wherein x is E R n And z.epsilon.R M To optimize the variables, a matrix ,/> , />The objective function consists of f (x) and g (z), from which it can be seen that the variables that were first disassembled are treated as different variables, and the constraint is also treated as such, and the iterative process is to alternately optimize the variables;
The augmented lagrangian function of equation (9) is shown below:
（10）
where λ is the dual variable and ρ is the coefficient of the quadratic penalty term;
the specific iteration steps of the ADMM algorithm are as follows:
（11）。
optionally, the step S3 specifically includes:
s31, all the participants reach consensus in advance, wherein the consensus comprises the number N of nodes and a penalty factor C, and each participant initializes v 0 and λ0 Setting a threshold epsilon i, wherein vi = [ω T , b i ] T ∈ R D+1 D is the feature quantity of the data set, v= [ V ] T , v T , . . . , v T ]Representing v i Is a collection of (3);
s32, all the participants and the adjacent nodes generate a multiplication triplet table of rho increment sequences and secret sharing multiplication calculation in advance, and simultaneously calculate matrix operation which is not changed in the iterative process in advance;
s33, in the t-th iteration, the participant i first interacts with all neighbor nodes j, e ij E, computing Σρ using FSCM-C privacy protection scheme t [v t + v t ]Then calculate Lagrangian multiplier；
S34, the party i updates v according to the formula (12) t+1 And broadcast to all neighbor nodes j, e ij ∈ E；
（12）
S35, the party i first interacts with all neighboring nodes j, e ij E, performing cooperative computing, and performing security computing by adopting FSCM-C privacy protection scheme, and updating lambda t+1 ；
S36, calculating a stop condition R t+1 If, as shown in formula (13)Stopping iteration, otherwise, returning to the step (3) to continue iteration;
（13）
After iterative training, all the participants obtain global optimal solutions, and then obtain global maximum interval classification functions。
Optionally, the FSCM-C privacy protection scheme specifically includes:
let [ N ]]As a collection of participants, B i = [X i , 1 i ]，1 i ∈ R M i，X i Representing a matrix of samples, Y i = diag ([y i1 , . . . , y iM ]) Diagonal matrix expressed as label, convenient for subsequent simplified model, xi i = [ξ i1 , . . . , ξ iM i ]A relaxation variable representing the ith-th party, C being a support vector machine penalty factor, and additionally, setting I D+1 Representing a unit vector with dimension D+1, N participants output a maximum interval linear classification function through a support vector machine model based on consensus, a network with N participants is assumed, a graph G= { N, E } represents a communication mode of the network, E represents a set of communication connection among the participants, and any node i epsilon [ N ]]Interact only with its neighboring nodes, the connection between the two nodes being denoted as e ij ∈ E ；
Participant(s)Local computation with neighbor node->Pre-generated parameters->Performing the calculation of a multiplication triplet table;
participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s) Local calculation +.>
Participant(s)Local meterCalculate->；
Both sides reconstruct and output the secret。
In another aspect, a distributed support vector machine training system based on a secret sharing mechanism is provided, the system comprising:
the decomposition module is used for decomposing the optimization problem of the support vector machine SVM into N sub-optimization problems by using an alternating direction multiplication method ADMM algorithm;
the local solving module is used for respectively carrying out local solving on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics;
and the training module is used for training and outputting global classifiers by using the ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters.
Optionally, the feature space of the input samples and labels is given in the SVM classification problemThe solved hyperplane is denoted +.>W is the normal vector of the hyperplane, b is the intercept, and the model of the SVM is:
（1）
where N is the number of samples, x i Represents the i-th sample, y i Represents the label corresponding to the ith sample, ζ i Is a relaxation variable, one for each sample, but the relaxation variables for points without outliers are all 0, indicating that the samples do not satisfyThe degree of constraint, fault tolerance is introduced, C is a penalty factor, and w and b are learned and output through a training method;
the Lagrangian function L of the solution model is shown in equation (2):
（2）
wherein alpha and mu are Lagrangian multipliers, alpha i ≥ 0， µ i ≥ 0；
The problem of the support vector machine is converted to solve the minimum and maximum problem of L by equation (2), with the goal of solving for the values of w, b and ζ that make L minimum by solving for a that makes L function maximum,the Lagrangian duality is applied here, the optimal solution of the original problem is solved by solving the dual problem, the solving difficulty can be reduced, the dual problem of the original problem is a maximum and minimum problem, w, b and ζ which enable the L function to be minimum are solved first, and then the maximum of alpha is solved for the L function, wherein the dual problem is shown as a formula (3):
（3）
wherein α= (α) 1 , α 2 , . . . , α N )
Is a Lagrangian multiplier vector;
in the model training stage, the training module is specifically configured to solve an optimization problem by using a steepest descent method in combination with an outlier penalty method, and the steps are as follows:
(1) External circulation
step 1: selecting an initial point alpha 0 Initial penalty factor M 1 =1, precision ε 1 > 0 , k := 1
step 2: to be used forFor an initial point, solving a constraint optimization problem as in equation (4):
（4）
solving F (alpha, M) by entering an inner circulation steepest descent method k ) Obtaining the minimum pointIs marked as->, wherein ：
（5）
step 3: if it isStopping the calculation to obtain the approximate minimum point +.>Otherwise let M k+1 =cM k Let k =k+1, turn step 1;
(2) Internal circulation
step 1: selection ofIs the initial point alpha of internal circulation 0 Precision epsilon 2 >0, let t: =1
step 2 determining the search direction for each alpha i And (3) calculating:
（6）
wherein ：
（7）
finally obtaining the search direction d t ：
（8）
step 3: from the slaveStarting from the edge d t Step length lambda of direction finding t
step 4: calculation of
step 5: if it isThe iteration is stopped, if not let t: =t+1, return step 2.
Alternatively, assuming n=2, the decomposition module is specifically configured to: the optimization problem of the SVM is decomposed into two sub-optimization problems f (x) and g (z):
（9）
wherein x is E R n and z ∈ RM To optimize the variables, a matrix ,/> , />The objective function consists of f (x) and g (z), from which it can be seen that the variables that were first disassembled are treated as different variables, and the constraint is also treated as such, and the iterative process is to alternately optimize the variables;
the augmented lagrangian function of equation (9) is shown below:
（10）
Where λ is the dual variable and ρ is the coefficient of the quadratic penalty term;
the specific iteration steps of the ADMM algorithm are as follows:
（11）。
optionally, the training module is specifically configured to:
all the participants reach consensus in advance, including the number of nodes N and penalty factor C, and each participant initializes v 0 and λ0 Setting a threshold epsilon i, wherein vi = [ω T , b i ] T ∈ R D+1 D is the feature quantity of the data set, v= [ V ] T , v T , . . . , v T ]Representing v i Is a collection of (3);
all the participants and the adjacent nodes generate multiplication triple table of rho increment sequence and secret sharing multiplication calculation in advance, and simultaneously calculate matrix operation which is not changed in the iterative process in advance;
in the t-th iteration, party i first interacts with all neighbor nodes j, e ij E, computing Σρ using FSCM-C privacy protection scheme t [v t + v t ]Then calculate Lagrangian multiplier；
Party i updates v according to equation (12) t+1 And broadcast to all neighbor nodes j, e ij ∈ E；
（12）
Participant i first interacts with all neighbor nodes j, e ij E, performing cooperative computing, and performing security computing by adopting FSCM-C privacy protection scheme, and updating lambda t+1 ；
Calculating a stop condition R t+1 If, as shown in formula (13)Stopping iteration, otherwise, returning to the step (3) to continue iteration;
（13）
after iterative training, all the participants obtain global optimal solutions, and then obtain global maximum interval classification functions 。
Optionally, the FSCM-C privacy protection scheme specifically includes:
let [ N ]]As a collection of participants, B i = [X i , 1 i ]，1 i ∈ R M i，X i Representing a matrix of samples, Y i = diag ([y i1 , . . . , y iM ]) Diagonal matrix expressed as label, convenient for subsequent simplified model, xi i = [ξ i1 , . . . , ξ iM i ]A relaxation variable representing the ith-th party, C being a support vector machine penalty factor, and additionally, setting I D+1 Representing a unit vector with dimension D+1, N participants output a maximum interval linear classification function through a support vector machine model based on consensus, a network with N participants is assumed, a graph G= { N, E } represents a communication mode of the network, E represents a set of communication connection among the participants, and any node i epsilon [ N ]]Interact only with its neighboring nodes, the connection between the two nodes being denoted as e ij ∈ E；
Participant(s)Neighbor node local computation +.>Pre-generated parameters->Performing the calculation of a multiplication triplet table;
participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)Local calculation +.>；
Participant(s)Local calculation +.>；
Both sides reconstruct and output the secret。
In another aspect, an electronic device is provided that includes a processor and a memory having at least one instruction stored therein that is loaded and executed by the processor to implement the above-described secret sharing mechanism based distributed support vector machine training method.
In another aspect, a computer readable storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement the above-described secret sharing mechanism based distributed support vector machine training method is provided.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:
the design of the invention constructs the SVM classifier in a distributed and privacy protection mode, the main idea is to combine the ADMM algorithm and the secret sharing technology, and utilize the decomposability of the ADMM algorithm and the property of secret sharing security calculation, and solve the optimization problem by combining the steepest descent method with the outlier punishment method in the model training stage, thereby realizing the aims of cooperative training and privacy protection of the complete distributed scene.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a training method of a distributed support vector machine based on a secret sharing mechanism according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a lateral distribution of data provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data longitudinal distribution provided by an embodiment of the present invention;
FIG. 4 is a flowchart of another training method of a distributed support vector machine based on a secret sharing mechanism according to an embodiment of the present invention;
FIG. 5 is a block diagram of a distributed support vector machine training system based on a secret sharing mechanism provided by an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.
The embodiment of the invention provides a distributed support vector machine training method based on a secret sharing mechanism, which can be realized by electronic equipment, wherein the electronic equipment can be a terminal or a server. A flowchart of a distributed support vector machine training method based on a secret sharing mechanism as shown in fig. 1, the process flow of the method may include the following steps:
S1, decomposing an optimization problem of a Support Vector Machine (SVM) into N sub-optimization problems by using an Alternating Direction Multiplication Method (ADMM) algorithm;
s2, respectively carrying out local solution on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics;
s3, training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters.
The traditional multi-data source support vector machine model is to collect the data of each data source together, then train a classification model by a central server, and the related security research work is all developed around the model. Assuming that there are M data samples, each sample containing multiple features, for the ith-th sample, there is x i ∈ R D And tag y i E { -1, 1}, let ζ represent pineRelaxation variables, which are obtained by solving a convex optimization problem, are shown in the following formula, to obtain a classification model, namely a maximum interval linear discriminant function。
In a real scene, the problems that a training sample set is huge and data is collected and stored in a distributed mode exist, and the communication cost of data transmission is increased and huge calculation load is brought to a central server by using a traditional multi-data-source support vector machine model. Therefore, the embodiment of the invention provides a complete distributed support vector machine model for collaborative training of different data holders realized by adopting an alternating direction multiplier method ADMM algorithm, is more suitable for a scene that data are collected and stored in a distributed mode, and can obtain more accuracy
In addition, the embodiment of the invention also provides a privacy protection scheme of the complete distributed support vector machine model. In the model training stage, the embodiment of the invention solves the optimization problem by combining a steepest descent method with an outlier penalty method.
The classification model and a training method of a distributed support vector machine based on a secret sharing mechanism according to the embodiment of the present invention are described in detail below with reference to fig. 2 to fig. 4, where the training method includes:
s1, decomposing an optimization problem of a Support Vector Machine (SVM) into N sub-optimization problems by using an Alternating Direction Multiplication Method (ADMM) algorithm;
optionally, the feature space of the input samples and labels is given in the SVM classification problemThe solved hyperplane is denoted +.>W is the normal vector of the hyperplane, b is the intercept, and the model of the SVM is:
（1）
where N is the number of samples, x i Represents the i-th sample, y i The label corresponding to the ith sample is represented, xi is a relaxation variable, each sample corresponds to one relaxation variable, but the relaxation variables of points without outliers are all 0, the degree that the samples do not meet the constraint is represented, fault tolerance is introduced, C is a penalty factor, and w and b are learned and output through a training method;
the Lagrangian function L of the solution model is shown in equation (2):
（2）
Wherein alpha and mu are Lagrangian multipliers, alpha i ≥ 0， µ i ≥ 0；
The problem of the support vector machine is converted to solve the minimum and maximum problem of L by equation (2), with the goal of solving for the values of w, b and ζ that make L minimum by solving for a that makes L function maximum,the Lagrangian duality is applied here, the optimal solution of the original problem is solved by solving the dual problem, the solving difficulty can be reduced, the dual problem of the original problem is a maximum and minimum problem, w, b and ζ which enable the L function to be minimum are solved first, and then the maximum of alpha is solved for the L function, wherein the dual problem is shown as a formula (3):
（3）
wherein α= (α) 1 , α 2 , . . . , α N ) Is a Lagrangian multiplier vector;
in the model training stage, the optimal problem is solved by combining a steepest descent method with an outlier penalty method, and the method comprises the following steps:
(1) External circulation
step 1: selecting an initial point alpha 0 Initial penalty factor M 1 =1, precision ε 1 > 0 , k := 1
step 2: to be used forFor an initial point, solving a constraint optimization problem as in equation (4):
（4）
solving F (alpha, M) by entering an inner circulation steepest descent method k ) Obtaining the minimum pointIs marked as->, wherein ：
（5）
step 3: if it isStopping the calculation to obtain the approximate minimum point +.>Otherwise let M k+1 =cM k Let k =k+1, turn step 1;
(2) Internal circulation
step 1: selection of Is the initial point alpha of internal circulation 0 Precision epsilon 2 >0, let t: =1
step 2 determining the search direction for each alpha i And (3) calculating:
（6）
wherein ：
（7）
finally obtaining the search direction d t ：
（8）
step 3: from the slaveStarting from the edge d t Step length lambda of direction finding t
step 4: calculation of
step 5: if it isThe iteration is stopped, if not let t: =t+1, return step 2.
Alternatively, assuming n=2, the S1 specifically includes: the optimization problem of the SVM is decomposed into two sub-optimization problems f (x) and g (z):
（9）
wherein x is E R n And z.epsilon.R M To optimize the variables, a matrix ,/> , />The objective function consists of f (x) and g (z) parts, from which it can be seen that the variables that were first broken down are seen as different variables, and the constraint is also treated as such, and the iterative process then proceeds with the variablesAlternately optimizing;
the augmented lagrangian function of equation (9) is shown below:
（10）
where λ is the dual variable and ρ is the coefficient of the quadratic penalty term;
the specific iteration steps of the ADMM algorithm are as follows:
（11）。
the ADMM algorithm is that the iteration of x and z is changed into independent iteration of x and z on the basis of the original augmentation Lagrangian algorithm, which means that the ADMM algorithm is divided into 2-blocks, and the ADMM algorithm can be popularized to N-blocks after deformation, namely the scenes of N participants.
S2, respectively carrying out local solution on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics;
When training samples of the data set are too large, the samples can also be segmented for collaborative training. Assuming there are N participants in a joint support vector machine (FSVM) scheme, there is M for the ith-th participant i Data sample, matrix X i Is a feature matrix, x ij Representing one sample data, y ij E { -1, 1} represents x ij Corresponding to the tag. In addition, the total number of features considered in the scheme of the embodiment of the invention is D.
When referring to a scenario where multiple data owners co-train a model, there are two types of data partitioning conditions, namely, partitioning by sample and by feature.
The first is divided by the number of samples, i.e., distributed laterally, as shown in fig. 2. This means that each party has all the characteristics and part of data, and the method is suitable for scenes in which multiple party data sources have the same sample characteristics, such as collaborative training of multiple e-commerce platforms, and better training effects are obtained by utilizing sharing of shopping data sets of different users with the same characteristic type to perform collaboration.
Thus X is i ∈ R Mi×D , x T ∈ R D Wherein (i=1, 2,) N and (j=1, 2,) M i )。
The second is by feature division, i.e. longitudinal distribution. As shown in fig. 3, it can be seen that each participant has the same amount of sample data, but the number or type of characteristics of the samples are not the same, such as different units or companies with different customer groups, such as banks, securities, homeowners, etc., desiring to cooperate to arrive at a global classifier that takes into account more comprehensive factors.
The embodiment of the invention is suitable for data sets which are distributed transversely, namely N participants respectively have the data sets, and the data sets have the same characteristics.
S3, training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters.
Security and privacy protection are very important propositions to consider for distributed systems, which will have a large amount of data and information to interact. Communication among users on different machines which are not trusted by each other is to ensure that sensitive information is not revealed, ensure confidentiality and prevent information from being stolen by malicious nodes.
There are two types of privacy protection schemes. The first is a method of data perturbation, and controllable noise is added to an intermediate state, so that differential privacy is satisfied, and differential privacy technology is mainly utilized. But the increased noise inevitably compromises the accuracy of the optimization results. The second method is a cryptographic method. But conventional cryptographic techniques cannot be applied directly in a fully distributed scenario without third party assistance. While there are two cryptographic methods that are suitable for fully distributed scenarios: secure multiparty computing technology and homomorphic encryption technology. Homomorphic encryption techniques are divided into full homomorphic encryption, which cannot be used for real-time optimization, and partial homomorphic encryption, which introduces a large amount of computational overhead. The generalized secure multiparty computing technology is that the addition in the ciphertext state is the same as the computation in the plaintext state, i.e. the processed data does not affect the final result that needs to be computed. Now generally to secure multiparty computing, we generally refer to Yao garbled circuits and secret sharing mechanisms. The embodiment of the invention discovers that the data disturbance method has high efficiency but can not provide strong security provable by cryptography, and can damage the accuracy of an optimized result, and the calculation cost and the communication cost of a multi-data source collaborative training scheme realized in a homomorphism are overlarge.
The idea of the secret sharing mechanism is to split one secret and distribute the split sub-secret to other participants, and only a suitable number of participants cooperate to recover the secret. The secret division is carried out by adopting a proper splitting means (the embodiment of the invention adopts addition secret division), after the sub-secrets are divided and distributed, each party can independently carry out local operation, and finally the calculation result obtained by reconstructing the secrets is consistent with the calculation result directly carried out by the undivided secrets, so that the method is suitable for multiparty split input to cooperatively carry out calculation, and the information of input data can not be revealed when the calculation result is obtained. The mathematical definition is as follows:wherein x represents the secret to be split, t represents the recovery threshold, n represents the number of splits, there is a recovery function R, for any m.gtoreq.t, there is +.>. I.e. dividing the secret into n shares, onlyAnd when the participant is greater than or equal to t, the secret can be reconstructed. Secret sharing is classified into boolean sharing and arithmetic sharing according to the type of computation, which correspond to the computation of boolean and arithmetic levels, respectively (boolean sharing is employed in the embodiments of the present invention).
As shown in fig. 4, in the training phase, N participants respectively solve the local sub-optimization problem, and perform interaction parameter solving global classifier by using ADMM algorithm. Further, the embodiment of the invention provides an FSVM-C privacy protection scheme for safely solving the global classifier by combining an ADMM algorithm and Boolean sharing and adding secret sharing.
The embodiment of the invention defines privacy as confidentiality of the intermediate state of the training process and the local support vector machine sub-problem. Because the ADMM needs to participate in parameter interaction among entities in a distributed scene, the embodiment of the invention protects the privacy of the objective function by protecting the interaction parameters.
Optionally, the step S3 specifically includes:
s31, all the participants reach consensus in advance, wherein the consensus comprises the number N of nodes and a penalty factor C, and each participant initializes v 0 and λ0 Setting a threshold epsilon i, wherein vi = [ω T , b i ] T ∈ R D+1 D is the feature quantity of the data set, v= [ V ] T , v T , . . . , v T ]Representing v i Is a collection of (3);
s32, all the participants and the adjacent nodes generate a multiplication triplet table of rho increment sequences and secret sharing multiplication calculation in advance, and simultaneously calculate matrix operation which is not changed in the iterative process in advance;
s33, in the t-th iteration, the participant i first interacts with all neighbor nodes j, e ij E, computing Σρ using FSCM-C privacy protection scheme t [v t + v t ]Then calculate Lagrangian multiplier；
S34, the party i updates v according to the formula (12) t+1 And broadcastTo all neighbor nodes j, e ij ∈ E；
（12）
S35, the party i first interacts with all neighboring nodes j, e ij E, performing cooperative computing, and performing security computing by adopting FSCM-C privacy protection scheme, and updating lambda t+1 ；
S36, calculating a stop condition R t+1 If, as shown in formula (13)Stopping iteration, otherwise, returning to the step (3) to continue iteration;
（13）
after iterative training, all the participants obtain global optimal solutions, and then obtain global maximum interval classification functions。
Optionally, the FSCM-C privacy protection scheme specifically includes:
let [ N ]]As a collection of participants, B i = [X i , 1 i ]，1 i ∈ R M i，X i Representing a matrix of samples, Y i = diag ([y i1 , . . . , y iM ]) Diagonal matrix expressed as label, convenient for subsequent simplified model, xi i = [ξ i1 , . . . , ξ iM i ]A relaxation variable representing the ith-th party, C being a support vector machine penalty factor, and additionally, setting I D+1 Representing a unit vector with dimension D+1, N participants output a maximum interval linear classification function through a support vector machine model based on consensus, a network with N participants is assumed, a graph G= { N, E } represents a communication mode of the network, E represents a set of communication connection among the participants, and any node i epsilon [ N ]]And only its neighbor nodesInteraction is performed and the connection between two nodes is denoted as e ij ∈ E；
Participant(s)Neighbor node local computation +.>Pre-generated parameters->Performing the calculation of a multiplication triplet table;
participant(s)For-> and />Performing addition secret segmentation;
participant(s) Send sub-secret to participant->；
Participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)Local calculation +.>；
Participant(s)Local calculation +.>；
Both sides reconstruct and output the secret。
The Algorithm is shown in Algorithm 1:
as shown in algorithm 1, for the t-th iteration, after the secret sharing technology is introduced by the aid of the respective sequences of the party i and the party j, interaction between nodes is increased, but many calculations requiring interaction, such as calculation of rho increment sequences and multiplication triples, can be pre-calculated, are independent of the iteration process, and do not increase calculation overhead of the training process. In addition, the calculation overhead of the secret sharing technology is far smaller than that of the homomorphic encryption technology. Although secret sharing is relatively inefficient compared to differential privacy, secret sharing techniques are chosen as the basis for the security scheme herein due to the advantages of high security and the high number of pre-calculations that increase the efficiency of the training process. Safely calculate ρ t+1 (v t+1 -v t+1 ) And then, continuing the iterative calculation of the optimization problem solving.
Privacy protection proof of embodiments of the present invention is demonstrated below:
assuming that the solution of the embodiment of the present invention is secure, all participants can only obtain the last global classifier during the whole solution execution process, and cannot obtain any additional information during the training process. The privacy protection of the scheme is proved below, and if an attacker of a real protocol exists, the ideal protocol and the real protocol are indistinguishable to the attacker, namely the information revealed by the real protocol does not exceed the rational protocol, the scheme can achieve the aim of privacy protection.
Privacy preserving evidence is analyzed using a generic combinable model. Assuming that the information obtained by the real protocol participant in the interactive execution scheme with other participants is V iew real [π, s]. Assuming a trusted simulator S, the user' S input can be simulated randomly to simulate the execution of a security training scheme, execute an ideal protocol, and use V iew ideal [π, s]Representing information acquired to perform the desired protocol. If the size of equation (14) is negligible, i.e., indistinguishable, then the protocol pi can be considered to have no additional data information revealed.
（14）
The training scheme of the FSVM-C privacy protection scheme of the embodiment of the invention relates to N participants. Assuming that there is a semi-honest attacker A, the other participants can be attacked to obtain the original data, here assuming that attacker A attacks N-1 participants P 1 , P 2 , . . . , P N-1 . The definition of security is such that attacker a can only know the data of the broken up participants and the last global classifier and not the information of the other participants.
The simulator S inputs training samples into the training scheme of the FSVM-C privacy protection scheme, and finally solves the global optimal solution, namely the maximum interval linear classification function, cooperatively. Firstly, as the training scheme of the FSVM-C privacy protection scheme does not need to interact with the original data, each participant solves the local optimization sub-problem locally by using a training sample, so that the information of the original data cannot be revealed. The simulator and the real protocol participants do not obtain any information. The aim of cooperatively updating training parameters of the support vector machine is achieved. The final simulator and the real protocol participants both obtain a global classifier g, letting the information obtained at this step be denoted T (g). The information obtained by the simulator and the real protocol participants is shown in the formula:
（15）
I.e. the real protocol and the simulator are indistinguishable. From the information view point of the simulator S and the information view point of the real protocol, the information revealed by the FSVM-C real protocol does not exceed the ideal protocol, so that the privacy protection scheme of the support vector machine with data distributed transversely can achieve the privacy protection target.
The embodiment of the invention provides an FSVM-C privacy protection scheme based on a secret sharing mechanism of a distributed support vector machine under a data transverse distribution scene, and the privacy protection of the original data and training intermediate state is realized in a pre-calculation stage and a training stage by combining an ADMM of a variable penalty matrix with a secret sharing technology. Meanwhile, the scheme safety is analyzed under an honest and curious safety model, and the scheme is proved to be capable of safely completing the aim of the multiparty collaborative training support vector machine.
As shown in fig. 5, an embodiment of the present invention further provides a distributed support vector machine training system based on a secret sharing mechanism, where the system includes:
a decomposition module 510, configured to decompose the optimization problem of the support vector machine SVM into N sub-optimization problems using an alternate direction multiplier ADMM algorithm;
the local solving module 520 is configured to solve the N sub-optimization problems locally by N participants, where the N participants each have a data set with data distributed laterally, and the data sets have the same characteristics;
And the training module 530 is used for training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters.
Optionally, the feature space of the input samples and labels is given in the SVM classification problemThe solved hyperplane is denoted +.>W is the normal vector of the hyperplane, b is the intercept, and the model of the SVM is:
（1）
where N is the number of samples, x i Represents the i-th sample, y i Represents the label corresponding to the ith sample, ζ i Is a relaxation variable, each sample corresponds to one relaxation variable, but the relaxation variables of points without outliers are all 0, which indicates the degree that the samples do not meet the constraint, fault tolerance is introduced, C is a penalty factor, and w and b are learned and output through a training method;
the Lagrangian function L of the solution model is shown in equation (2):
（2）
wherein alpha and mu are Lagrangian multipliers, alpha i ≥ 0， µ i ≥ 0；
The problem of the support vector machine is converted to solve the minimum and maximum problem of L by equation (2), with the goal of solving for the values of w, b and ζ that make L minimum by solving for a that makes L function maximum, The Lagrangian duality is applied here, the optimal solution of the original problem is solved by solving the dual problem, the solving difficulty can be reduced, the dual problem of the original problem is a maximum and minimum problem, w, b and ζ which enable the L function to be minimum are solved first, and then the maximum of alpha is solved for the L function, wherein the dual problem is shown as a formula (3):
（3）
wherein α= (α) 1 , α 2 , . . . , α N ) Is a Lagrangian multiplier vector;
in the model training stage, the optimal problem is solved by combining a steepest descent method with an outlier penalty method, and the method comprises the following steps:
(1) External circulation
step 1: selecting an initial point alpha 0 Initial penalty factor M 1 =1, precision ε 1 > 0 , k := 1
step 2: to be used forFor an initial point, solving a constraint optimization problem as in equation (4):
（4）/>
solving F (alpha, M) by entering an inner circulation steepest descent method k ) Obtaining the minimum pointIs marked as->, wherein ：
（5）
step 3: if it isStopping the calculation to obtain the approximate minimum point +.>Otherwise let M k+1 =cM k Let k =k+1, turn step 1;
(2) Internal circulation
step 1: selection ofIs the initial point alpha of internal circulation 0 Precision epsilon 2 >0, let t: =1
step 2 determining the search direction for each alpha i And (3) calculating:
（6）
wherein ：
（7）
finally obtaining the search direction d t ：
（8）
step 3: from the slaveStarting from the edge d t Step length lambda of direction finding t
step 4: calculation of
step 5: if it is The iteration is stopped, if not let t: =t+1, return step 2.
Alternatively, assuming n=2, the S1 specifically includes: the optimization problem of the SVM is decomposed into two sub-optimization problems f (x) and g (z):
（9）
wherein x is E R n And z.epsilon.R M To optimize the variables, a matrix ,/> , />The objective function consists of f (x) and g (z), from which it can be seen that the variables that were first disassembled are treated as different variables, and the constraint is also treated as such, and the iterative process is to alternately optimize the variables;
the augmented lagrangian function of equation (9) is shown below:
（10）
where λ is the dual variable and ρ is the coefficient of the quadratic penalty term;
the specific iteration steps of the ADMM algorithm are as follows:
（11）。
optionally, the training module is specifically configured to:
all the participants reach consensus in advance, including the number of nodes N and penalty factor C, and each participant initializes v 0 and λ0 Setting a threshold epsilon i, wherein vi = [ω T , b i ] T ∈ R D+1 D is the feature quantity of the data set, v= [ V ] T , v T , . . . , v T ]Representing v i Is a collection of (3);
all the participants and the adjacent nodes generate multiplication triple table of rho increment sequence and secret sharing multiplication calculation in advance, and simultaneously calculate matrix operation which is not changed in the iterative process in advance;
In the t-th iteration, party i first interacts with all neighbor nodes j, e ij E, computing Σρ using FSCM-C privacy protection scheme t [v t + v t ]Then calculate the Lagrangian multiplicationSon；
Party i updates v according to equation (12) t+1 And broadcast to all neighbor nodes j, e ij ∈ E；
（12）
Participant i first interacts with all neighbor nodes j, e ij E, performing cooperative computing, and performing security computing by adopting FSCM-C privacy protection scheme, and updating lambda t+1 ；
Calculating a stop condition R t+1 Stopping iteration if the iteration is stopped as shown in the formula (13), otherwise, returning to the step (3) to continue iteration;
（13）
after iterative training, all the participants obtain global optimal solutions, and then obtain global maximum interval classification functions。
Optionally, the FSCM-C privacy protection scheme specifically includes:
let [ N ]]As a collection of participants, B i = [X i , 1 i ]，1 i ∈ R M i，X i Representing a matrix of samples, Y i = diag ([y i1 , . . . , y iM ]) Diagonal matrix expressed as label, convenient for subsequent simplified model, xi i = [ξ i1 , . . . , ξ iM i ]A relaxation variable representing the ith-th party, C being a support vector machine penalty factor, and additionally, setting I D+1 The unit vector with dimension D+1 is represented, N participants output the maximum interval linear classification function through a support vector machine model based on consensus, a network with N participants is assumed, the communication mode of the network is represented by a graph G= { N, E }, and E table Illustrating a collection of communication connections between participants, arbitrary node i e N]Interact only with its neighboring nodes, the connection between the two nodes being denoted as e ij ∈ E；
Participant(s)Local computation with neighbor node->Pre-generated parameters->Performing the calculation of a multiplication triplet table;
participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)For-> and />Performing row addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)Local calculation +.>；
Participant(s)Local calculation +.>；
Both sides reconstruct and output the secret。
The functional structure of the distributed support vector machine training system based on the secret sharing mechanism provided by the embodiment of the invention corresponds to the distributed support vector machine training method based on the secret sharing mechanism provided by the embodiment of the invention, and is not described herein.
Fig. 6 is a schematic structural diagram of an electronic device 600 according to an embodiment of the present invention, where the electronic device 600 may have a relatively large difference due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 and one or more memories 602, where at least one instruction is stored in the memories 602, and the at least one instruction is loaded and executed by the processors 601 to implement the steps of the above-described distributed support vector machine training method based on a secret sharing mechanism.
In an exemplary embodiment, a computer readable storage medium, such as a memory comprising instructions executable by a processor in a terminal to perform the above-described secret sharing mechanism based distributed support vector machine training method is also provided. For example, the computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (2)

1. A distributed support vector machine training method based on a secret sharing mechanism, the method comprising:
S1, decomposing an optimization problem of a Support Vector Machine (SVM) into N sub-optimization problems by using an Alternating Direction Multiplication Method (ADMM) algorithm;
s2, respectively carrying out local solution on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics;
s3, training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters;
feature space given input samples and labels in SVM classification problemThe solved hyperplane is denoted +.>W is the normal vector of the hyperplane, b is the intercept, and the model of the SVM is:
（1）
where N is the number of samples, x i Represents the i-th sample, y i Represents the label corresponding to the ith sample, ζ i Is a relaxation variable, each sample corresponds to one relaxation variable, but the relaxation variables of points without outliers are all 0, which indicates the degree that the samples do not meet the constraint, fault tolerance is introduced, C is a penalty factor, and w and b are learned and output through a training method;
the Lagrangian function L of the solution model is shown in equation (2):
（2）
Wherein alpha and mu are Lagrangian multipliers, alpha i ≥ 0，µ i ≥ 0；
The problem of the support vector machine is converted to solve the minimum and maximum problem of L by equation (2), with the goal of solving for the values of w, b and ζ that make L minimum by solving for a that makes L function maximum,the Lagrangian duality is applied here, the optimal solution of the original problem is solved by solving the dual problem, the solving difficulty can be reduced, the dual problem of the original problem is a maximum and minimum problem, w, b and ζ which enable the L function to be minimum are solved first, and then the maximum of alpha is solved for the L function, wherein the dual problem is shown as a formula (3):
（3）
wherein α= (α 1 , α 2 , . . . , α N ) Is a Lagrangian multiplier vector;
in the model training stage, the optimal problem is solved by combining a steepest descent method with an outlier penalty method, and the method comprises the following steps:
(1) External circulation
step 1: selecting an initial point alpha 0 Initial penalty factor M 1 =1, precision ε 1 > 0 , k := 1
step 2: to be used forFor an initial point, solving a constraint optimization problem as in equation (4):
（4）
solving F (alpha, M) by entering an inner circulation steepest descent method k ) Obtaining the minimum pointIs marked as->, wherein ：
（5）
step 3: if it isStopping the calculation to obtain the approximate minimum point +.>Otherwise letM k+1 =cM k Let k =k+1, turn step 1;
(2) Internal circulation
step 1: selection of Is the initial point alpha of internal circulation 0 Precision ofε 2 >0, let t: =1
step 2 determining the search direction for eachα i And (3) calculating:
（6）
wherein ：
（7）
finally obtaining the search directiond t ：
（8）
step 3: from the slaveStarting from the edged t Step length of direction calculationλ t
step 4: calculation of；
step 5: if it isStopping iteration, if not, letting t be =t+1, and returning to step 2;
assuming n=2, the S1 specifically includes: decomposing the optimization problem of SVM into two sub-optimization problemsf (x) and g (z)：
（9）
wherein ,x ∈ R n andz ∈ R M to optimize the variables, a matrixA ∈R s∗n , B ∈R s∗M , c ∈R s The objective function is composed off (x) and g (z) Two-part construction, from this form, the first-to-break variable is seen asDifferent variables and constraint conditions are treated as such, and the iterative process is to alternately optimize the variables;
the augmented lagrangian function of equation (9) is shown below:
（10）
where λ is the dual variable and ρ is the coefficient of the quadratic penalty term;
the specific iteration steps of the ADMM algorithm are as follows:
（11）；
the step S3 specifically comprises the following steps:
s31, all the participants reach consensus in advance, including the number of nodes N and penalty factor C, and each participant initializesv 0 Andλ 0 setting a threshold valueε i , wherein v i = [ω T , b i ] T ∈ R D+1 D is the characteristic quantity of the data set,V = [v T , v T , . . . , v T ]representation ofv i Is a collection of (3);
s32, all the participants and the adjacent nodes generate a multiplication triplet table of rho increment sequences and secret sharing multiplication calculation in advance, and simultaneously calculate matrix operation which is not changed in the iterative process in advance;
S33, in the t-th iteration, the participant i first interacts with all neighbor nodesj, e ij ∈ ESigma computation using FSCM-C privacy preserving schemeρ t [v t + v t ]Then calculate Lagrangian multiplier；
S34、Party i updates according to equation (12)v t+1 And broadcast to all neighbor nodesj, e ij ∈ E；
（12）
S35, the party i first and all neighbor nodesj, e ij ∈ ECollaborative computing is performed, where security computing is also performed using FSCM-C privacy preserving scheme, and then updatedλ t+1 ；
S36, calculating a stop condition R t+1 If, as shown in formula (13)Stopping iteration, otherwise, returning to the step (3) to continue iteration;
（13）
after iterative training, all the participants obtain global optimal solutions, and then obtain global maximum interval classification functions；
The FSCM-C privacy protection scheme specifically comprises the following steps:
let [ N ]]For a collection of parties to be considered,B i = [X i , 1 i ]，1 i ∈R M i，X i representing a matrix of samples,Y i = diag ([y i1 , . . . , y iM ]) Represented as a diagonal matrix of labels, facilitating subsequent simplified models,ξ i = [ξ i1 , . . . , ξ iM i]a relaxation variable representing the ith-th party, C being a support vector machine penalty factor, additionally, settingI D+1 Representing a unit vector with dimension D+1, N participants output a maximum interval linear classification function through a support vector machine model based on consensus, a network with N participants is assumed, a graph G= { N, E } represents a communication mode of the network, E represents a set of communication connection among the participants, and any node i epsilon [ N ] ]Interact with only its neighboring nodes, the connection between the two nodes being represented ase ij ∈ E；
Participant(s)Local computation with neighbor node->Pre-generated parameters->Performing the calculation of a multiplication triplet table;
participant(s)For-> and />Performing row addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)Local calculation +.>；
Participant(s)Local calculation +.>；
Both sides reconstruct and output the secret。
2. A distributed support vector machine training system based on a secret sharing mechanism, the system comprising:
the decomposition module is used for decomposing the optimization problem of the support vector machine SVM into N sub-optimization problems by using an alternating direction multiplication method ADMM algorithm;
the local solving module is used for respectively carrying out local solving on the N sub-optimization problems by N participants, wherein the N participants respectively have data sets with data distributed transversely, and the data sets have the same characteristics;
the training module is used for training and outputting global classifiers by using ADMM algorithm interaction parameters by the N participants, and protecting the parameters by adopting an FSVM-C privacy protection scheme based on a secret sharing mechanism when the N participants use the ADMM algorithm interaction parameters;
Feature space given input samples and labels in SVM classification problemThe solved hyperplane is denoted +.>W is the normal vector of the hyperplane, b is the intercept, and the model of the SVM is:
（1）
where N is the number of samples and,x i the i-th sample is represented by the expression,y i the label corresponding to the ith sample is represented, xi is a relaxation variable, each sample corresponds to one relaxation variable, but the relaxation variables of points without outliers are all 0, the degree that the samples do not meet the constraint is represented, fault tolerance is introduced, C is a penalty factor, and w and b are learned and output through a training method;
the Lagrangian function L of the solution model is shown in equation (2):
（2）
where alpha and mu are both lagrange multipliers,α i ≥ 0，µ i ≥ 0；
the problem of the support vector machine is converted to solve the minimum and maximum problem of L by equation (2), with the goal of solving for the values of w, b and ζ that make L minimum by solving for a that makes L function maximum,where the application isThe Lagrangian duality is achieved by solving the optimal solution of the original problem, the solving difficulty can be reduced, the dual problem of the original problem is the maximum and minimum problem, w, b and ζ which enable the L function to be minimum are solved first, and then the maximum of alpha is solved for the L function, wherein the dual problem is shown in the formula (3):
（3）
wherein α= (α 1 , α 2 , . . . , α N ) Is a Lagrangian multiplier vector;
in the model training stage, the training module is specifically configured to solve an optimization problem by using a steepest descent method in combination with an outlier penalty method, and the steps are as follows:
(1) External circulation
step 1: selecting an initial point alpha 0 Initial penalty factor M 1 =1, precision ε 1 > 0 , k := 1
step 2: alpha is alpha k−1 For an initial point, solving a constraint optimization problem as in equation (4):
（4）
solving F (alpha, M) by entering an inner circulation steepest descent method k ) Obtaining the minimum pointIs marked as->, wherein ：
（5）
step 3: if it isStop the meterCalculating to obtain approximate minimum point->Otherwise letM k+1 =cM k Let k =k+1, turn step 1;
(2) Internal circulation
step 1: selection ofIs the initial point alpha of internal circulation 0 Precision ofε 2 >0, let t: =1
step 2 determining the search direction for eachα i And (3) calculating:
（6）
wherein ：
（7）
finally obtaining the search directiond t ：
（8）
step 3: from the slaveStarting from the edged t Step length of direction calculationλ t
step 4: calculation ofstep 5: if->Stopping iteration, if not, letting t be =t+1, and returning to step 2;
assuming n=2, the decomposition module is specifically configured to: decomposing the optimization problem of SVM into two sub-optimization problemsf (x) and g (z)：
（9）
wherein ,x ∈ R n andz ∈ R M to optimize the variables, a matrixA ∈ R s∗n , B ∈ R s∗M , c ∈ R s The objective function is composed off (x) and g (z) Two parts are formed, from the form, the variables which are disassembled firstly are regarded as different variables, the constraint condition is treated as the same, and the iterative process is to alternately optimize the variables;
The augmented lagrangian function of equation (9) is shown below:
（10）
where λ is the dual variable and ρ is the coefficient of the quadratic penalty term;
the specific iteration steps of the ADMM algorithm are as follows:
（11）；
the training module is specifically configured to:
all the participants reach consensus in advance, including node number N and penalty factor C, each participant is initializedv 0 Andλ 0 setting a threshold valueε i , wherein v i = [ω T , b i ] T ∈ R D+1 D is the characteristic quantity of the data set,V = [v T , v T , . . . , v T ]watch (watch)Showing thev i Is a collection of (3);
all the participants and the adjacent nodes generate multiplication triple table of rho increment sequence and secret sharing multiplication calculation in advance, and simultaneously calculate matrix operation which is not changed in the iterative process in advance;
in the t-th iteration, party i first interacts with all neighbor nodesj, e ij ∈ ESigma computation using FSCM-C privacy preserving schemeρ t [v t + v t ]Then calculate Lagrangian multiplier；
Party i updates according to equation (12)v t+1 And broadcast to all neighbor nodesj, e ij ∈ E；
（12）
Participant i first interacts with all neighbor nodesj, e ij ∈ ECollaborative computing is performed, where security computing is also performed using FSCM-C privacy preserving scheme, and then updatedλ t+1 ；
Calculating a stop condition R t+1 If, as shown in formula (13)Stopping iteration, otherwise, returning to the step (3) to continue iteration;
（13）
after iterative training, all the participants obtain global optimal solutions, and then obtain global maximum interval classification functions ；
The FSCM-C privacy protection scheme specifically comprises the following steps:
let [ N ]]For a collection of parties to be considered,B i = [X i , 1 i ]，1 i ∈ R M i，X i representing a matrix of samples,Y i = diag ([y i1 , . . . , y iM ]) Represented as a diagonal matrix of labels, facilitating subsequent simplified models,ξ i = [ξ i1 , . . . , ξ iM i ]a relaxation variable representing the ith-th party, C being a support vector machine penalty factor, additionally, settingI D+1 Representing a unit vector with dimension D+1, N participants output a maximum interval linear classification function through a support vector machine model based on consensus, a network with N participants is assumed, a graph G= { N, E } represents a communication mode of the network, E represents a set of communication connection among the participants, and any node i epsilon [ N ]]Interact with only its neighboring nodes, the connection between the two nodes being represented ase ij ∈ E ；
Participant(s)Local computation with neighbor node->Generating parameters in advance to calculate a multiplication triplet table;
participant(s)For-> and />Performing row addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)For-> and />Performing addition secret segmentation;
participant(s)Send sub-secret to participant->；
Participant(s)Local calculation +.>；
Participant(s)Local calculation +.>；
Both sides reconstruct and output the secret。
CN202310869601.5A 2023-07-17 2023-07-17 Secret sharing mechanism-based distributed support vector machine training method and system Active CN116595589B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310869601.5A CN116595589B (en) 2023-07-17 2023-07-17 Secret sharing mechanism-based distributed support vector machine training method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310869601.5A CN116595589B (en) 2023-07-17 2023-07-17 Secret sharing mechanism-based distributed support vector machine training method and system

Publications (2)

Publication Number Publication Date
CN116595589A CN116595589A (en) 2023-08-15
CN116595589B true CN116595589B (en) 2023-10-10

Family

ID=87612038

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310869601.5A Active CN116595589B (en) 2023-07-17 2023-07-17 Secret sharing mechanism-based distributed support vector machine training method and system

Country Status (1)

Country Link
CN (1) CN116595589B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106371610A (en) * 2016-09-23 2017-02-01 重庆金瓯科技发展有限责任公司 Method for detecting driving fatigue based on electroencephalogram
CN113627485A (en) * 2021-07-10 2021-11-09 南京理工大学 Unbalanced big data distributed classification method based on ADMM
CN114925786A (en) * 2022-07-08 2022-08-19 蓝象智联(杭州)科技有限公司 Longitudinal federal linear support vector classification method based on secret sharing
CN115225516A (en) * 2022-07-18 2022-10-21 重庆邮电大学 Least square support vector machine network flow prediction method based on improved ABC-VMD

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8719194B2 (en) * 2011-09-19 2014-05-06 Siemens Aktiengesellschaft Hybrid interior-point alternating directions algorithm for support vector machines and feature selection

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106371610A (en) * 2016-09-23 2017-02-01 重庆金瓯科技发展有限责任公司 Method for detecting driving fatigue based on electroencephalogram
CN113627485A (en) * 2021-07-10 2021-11-09 南京理工大学 Unbalanced big data distributed classification method based on ADMM
CN114925786A (en) * 2022-07-08 2022-08-19 蓝象智联(杭州)科技有限公司 Longitudinal federal linear support vector classification method based on secret sharing
CN115225516A (en) * 2022-07-18 2022-10-21 重庆邮电大学 Least square support vector machine network flow prediction method based on improved ABC-VMD

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
垂直分布数据的隐私保护支持向量机;刘晓红 等;佳木斯大学学报(自然科学版);第29卷(第03期);全文 *
面向SVM的隐私保护方法研究进展;彭晓冰 等;江苏大学学报(自然科学版);第38卷(第01期);全文 *

Also Published As

Publication number Publication date
CN116595589A (en) 2023-08-15

Similar Documents

Publication Publication Date Title
Chaudhari et al. Trident: Efficient 4pc framework for privacy preserving machine learning
Chen et al. Vertically federated graph neural network for privacy-preserving node classification
Zhu et al. From federated learning to federated neural architecture search: a survey
Li et al. Multi-key privacy-preserving deep learning in cloud computing
Zhang et al. GELU-Net: A Globally Encrypted, Locally Unencrypted Deep Neural Network for Privacy-Preserved Learning.
Zhao et al. PVD-FL: A privacy-preserving and verifiable decentralized federated learning framework
CN111242290B (en) Lightweight privacy protection generation countermeasure network system
Li et al. Optimizing privacy-preserving outsourced convolutional neural network predictions
Mendis et al. A blockchain-powered decentralized and secure computing paradigm
Xu et al. Nn-emd: Efficiently training neural networks using encrypted multi-sourced datasets
Erkin et al. Privacy-preserving distributed clustering
CN115660050A (en) Robust federated learning method with efficient privacy protection
Yang et al. Lightweight privacy-preserving GAN framework for model training and image synthesis
Pramkaew et al. Lightweight scheme of secure outsourcing SVD of a large matrix on cloud
CN112532383A (en) Privacy protection calculation method based on secret sharing
Song et al. Sok: Training machine learning models over multiple sources with privacy preservation
Hao et al. Fastsecnet: An efficient cryptographic framework for private neural network inference
Zhang et al. SecureTrain: An approximation-free and computationally efficient framework for privacy-preserved neural network training
CN116388954B (en) General secret state data security calculation method
Cao et al. Privacy-preserving healthcare monitoring for IoT devices under edge computing
Liu et al. ESA-FedGNN: Efficient secure aggregation for federated graph neural networks
Deng et al. Non-interactive and privacy-preserving neural network learning using functional encryption
CN116595589B (en) Secret sharing mechanism-based distributed support vector machine training method and system
CN116975906A (en) Ridge regression privacy protection algorithm based on secure multiparty calculation
Li et al. FPCNN: A fast privacy-preserving outsourced convolutional neural network with low-bandwidth

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant