CN112001501B - Parameter updating method, device and equipment of AI distributed training system - Google Patents
Parameter updating method, device and equipment of AI distributed training system Download PDFInfo
- Publication number
- CN112001501B CN112001501B CN202010820131.XA CN202010820131A CN112001501B CN 112001501 B CN112001501 B CN 112001501B CN 202010820131 A CN202010820131 A CN 202010820131A CN 112001501 B CN112001501 B CN 112001501B
- Authority
- CN
- China
- Prior art keywords
- target
- node
- updating
- worker node
- matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 title claims abstract description 118
- 238000000034 method Methods 0.000 title claims abstract description 87
- 239000011159 matrix material Substances 0.000 claims abstract description 106
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 57
- 230000001133 acceleration Effects 0.000 claims abstract description 22
- 238000004590 computer program Methods 0.000 claims description 10
- 150000001875 compounds Chemical class 0.000 claims description 4
- 238000004891 communication Methods 0.000 abstract description 11
- 238000013473 artificial intelligence Methods 0.000 description 99
- 238000010586 diagram Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000012935 Averaging Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Abstract
The application discloses a parameter updating method of an AI distributed training system, which comprises the following steps: starting a training task of an AI algorithm model on a target worker node of a distributed heterogeneous system, controlling the node to load model parameters, randomly selecting sample data of a kth iterative training for the node, carrying out gradient updating on the model parameters, randomly creating a target node set, carrying out non-zero value updating on an adjacent matrix by using the set, and updating model parameters on each node in the set by using the updated adjacent matrix; and when the kth iterative training is finished, if the AI algorithm model is converged, repeating the iterative training on the node until the node finishes M iterative training, and judging that the distributed heterogeneous system finishes the AI acceleration task. By the method, the requirements of each worker node in the distributed computing cluster on communication bandwidth during parameter synchronization can be reduced while a mixed heterogeneous distributed computing environment is supported.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for updating parameters of an AI distributed training system.
Background
In practical application, a distributed cluster is often used to accelerate a training task of an AI (Artificial Intelligence) algorithm model, when a plurality of worker nodes in the distributed cluster are used to perform data parallel training on the AI algorithm model, firstly, the same AI algorithm model is deployed on each worker node, and batch-time iteration processing is performed on the marked training data, wherein in each iteration process, a batch of training data needs to be divided into N micro-batches according to the number of the worker nodes, then, the N micro-batches of training data are distributed to different worker nodes to perform model training, and finally, after all the worker nodes complete training of each micro-batch of training data, model parameters on each worker node are synchronously updated.
At present, parameter synchronization updating methods of an AI distributed training system mainly comprise a Parameter-Server algorithm, an All-Reduce algorithm and a Ring-All-Reduce algorithm, and because the three Parameter synchronization methods are All oriented to isomorphic distributed computing environments, all worker nodes in the distributed computing system are required to be provided with completely same computing equipment, and All worker nodes are required to adopt communication links with the same bandwidth, so that the linear acceleration ratio of the whole distributed computing system can be improved. However, in practical applications, various new computing devices are continuously added to worker nodes of the AI distributed training system, and in this case, if the parameters in the AI distributed training system are updated by using the above three algorithms, the overall performance of the AI distributed training system is limited to the worker node with the slowest computing performance or the communication link with the slowest transmission in the distributed computing environment. At present, no effective solution exists for the technical problem.
Therefore, the technical problem to be solved by the technical staff in the field is how to enable the AI distributed training system to support a mixed heterogeneous distributed computing environment and reduce the requirement of each worker node in the distributed computing cluster on communication bandwidth during parameter synchronization.
Disclosure of Invention
In view of this, an object of the present invention is to provide a method, an apparatus, a device, and a medium for updating parameters of an AI distributed training system, so that the AI distributed training system can support a hybrid heterogeneous distributed computing environment and can also reduce a requirement for a communication bandwidth when each worker node in a distributed computing cluster performs parameter synchronization. The specific scheme is as follows:
a parameter updating method of an AI distributed training system comprises the following steps:
when the distributed heterogeneous system needs to finish an AI acceleration task, starting a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system, and initializing model parameters and an adjacent matrix of the target worker node; an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system;
controlling the target worker node to load preset target model parameters, and randomly selecting sample data of kth iterative training for the target worker node; wherein k is more than or equal to 1;
gradient updating is carried out on the target model parameters based on the sample data, and a target node set is randomly created for the target worker nodes;
updating a non-zero value of the adjacent matrix by using the target node set to obtain an updated adjacent matrix, and updating model parameters on each worker node in the target node set by using the updated adjacent matrix;
when the target worker node finishes the kth iterative training, judging whether an AI algorithm model on the target worker node converges;
if yes, the target worker node is judged to complete the kth AI algorithm model training task, the step of controlling the target worker node to load preset target model parameters is repeatedly executed, and the distributed heterogeneous system is judged to complete the AI acceleration task until the target worker node completes M times of iterative training; wherein M is more than or equal to 2, and M is the preset iteration number.
Preferably, the method further comprises the following steps:
and constructing the distributed heterogeneous system by using a plurality of worker nodes provided with the GPU and/or AI chips and/or FPGA.
Preferably, after the process of judging whether the AI algorithm model on the target worker node converges, the method further includes:
and if not, re-executing the step of controlling the target worker node to load preset target model parameters and randomly selecting sample data of the kth iterative training for the target worker node.
Preferably, the process of initializing the model parameters and the adjacency matrix of the target worker node includes:
and initializing the adjacent matrix of the target worker node to be a zero matrix E.
Preferably, the process of updating the adjacency matrix with a non-zero value by using the target node set to obtain an updated adjacency matrix includes:
carrying out non-zero value updating on the zero matrix E by utilizing a node set g to obtain an updated adjacent matrix;
wherein the expression of updating the adjacency matrix is as follows:
in the formula, g is the node set, i and j are the row number and the column number in the zero matrix E, respectively, and u represents the row number and the column number when i and j are the same.
Preferably, the process of updating the model parameters on each worker node in the target node set by using the updated adjacency matrix includes:
updating the model parameters on each worker node in the target node set by using the updated adjacency matrix to obtain updated parameters;
wherein the expression of the update parameter is:
in the formula (I), the compound is shown in the specification,
a parameter matrix updated based on the parameters of the node set g in the k iteration process,
and updating the adjacency matrix based on the node set g in the k iteration process.
Correspondingly, the invention also discloses a parameter updating device of the AI distributed training system, which comprises the following components:
the model initialization module is used for starting a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system when the distributed heterogeneous system needs to finish an AI acceleration task, and initializing model parameters and an adjacent matrix of the target worker node; an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system;
the sample selection module is used for controlling the target worker node to load preset target model parameters and randomly selecting sample data of the kth iterative training for the target worker node; wherein k is more than or equal to 1;
the gradient updating module is used for carrying out gradient updating on the target model parameters based on the sample data and randomly creating a target node set for the target worker node;
the parameter updating module is used for updating a nonzero value of the adjacent matrix by using the target node set to obtain an updated adjacent matrix and updating model parameters on each worker node in the target node set by using the updated adjacent matrix;
the convergence judging module is used for judging whether an AI algorithm model on the target worker node converges or not when the target worker node completes the kth iterative training;
the task training module is used for judging that the target worker node completes the kth AI algorithm model training task when the judgment result of the convergence judgment module is yes, repeatedly executing the step of controlling the target worker node to load preset target model parameters until the target worker node completes M times of iterative training, and judging that the distributed heterogeneous system completes the AI acceleration task; wherein M is more than or equal to 2, and M is the preset iteration number.
Correspondingly, the invention also discloses a parameter updating device of the AI distributed training system, which comprises:
a memory for storing a computer program;
a processor for implementing a parameter updating method of an AI distributed training system as disclosed in the foregoing when executing the computer program.
Accordingly, the present invention also discloses a computer readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the parameter updating method of the AI distributed training system as disclosed in the foregoing.
Therefore, in the invention, when the distributed heterogeneous system needs to complete the AI acceleration task, an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system, and model parameters and an adjacent matrix of a target worker node are initialized; then, controlling a target worker node to load preset target model parameters, and randomly selecting sample data of the kth iterative training for the target worker node; and then, carrying out gradient updating on target model parameters based on sample data, randomly creating a target node set for target worker nodes, after the target node set is created, carrying out non-zero value updating on an adjacent matrix by using the target node set to obtain an updated adjacent matrix, updating model parameters on each worker node in the target node set by using the updated adjacent matrix, judging whether an AI algorithm model on the target worker nodes is converged when the target worker nodes finish the kth iterative training, if the IA algorithm model on the target worker nodes is converged, indicating that the target worker nodes finish the kth AI algorithm model training task, and finally, repeatedly executing the step of controlling the target worker nodes to load preset target model parameters until the target worker nodes finish M iterative training, indicating that the distributed heterogeneous system finishes the AI acceleration task. Obviously, in the parameter synchronization method provided by the invention, because the AI acceleration task is performed based on the distributed heterogeneous system, and in the parameter synchronization process of the AI model, the target worker node randomly creates a target node set and only updates the parameters of the randomly selected worker node, the method can enable the AI distributed training system to support the mixed heterogeneous distributed computing environment and simultaneously reduce the requirement of each worker node in the distributed computing cluster on the communication bandwidth during the parameter synchronization. Correspondingly, the parameter updating device, the equipment and the medium of the AI distributed training system provided by the invention also have the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a parameter updating method of an AI distributed training system according to an embodiment of the present invention;
fig. 2 is a structural diagram of a distributed heterogeneous system according to an embodiment of the present invention;
fig. 3 is a structural diagram of a parameter updating apparatus of an AI distributed training system according to an embodiment of the present invention;
fig. 4 is a structural diagram of a parameter updating device of an AI distributed training system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a parameter updating method of an AI distributed training system according to an embodiment of the present invention, where the parameter updating method includes:
step S11: when the distributed heterogeneous system needs to finish an AI acceleration task, starting a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system, and initializing model parameters and an adjacent matrix of the target worker node;
an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system;
in this embodiment, a novel parameter updating method for an AI distributed training system is provided, and the method can also reduce the requirement on communication bandwidth when parameter synchronization is performed on each worker node in a distributed computing cluster while supporting a heterogeneous distributed computer environment.
Specifically, in this embodiment, an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system, and when the distributed heterogeneous system needs to complete an AI acceleration task, a training task of the AI algorithm model is started on a target worker node of the distributed heterogeneous system, and model parameters and an adjacency matrix of the target worker node are initialized. It should be noted that the target worker node refers to any one of the computing nodes in the distributed heterogeneous system, and in this embodiment, the number of worker nodes in the distributed heterogeneous system is not limited.
Referring to fig. 2, fig. 2 is a structural diagram of a distributed heterogeneous system according to an embodiment of the present invention, where there are 6 worker nodes in the distributed heterogeneous system, and in an actual application, the distributed heterogeneous system may be represented as G = { V, E }, where V represents each work node in the distributed heterogeneous system, and E represents an adjacent matrix between each worker node in the distributed heterogeneous system.
Step S12: controlling a target worker node to load preset target model parameters, and randomly selecting sample data of a kth iterative training for the target worker node; wherein k is more than or equal to 1;
step S13: gradient updating is carried out on target model parameters based on sample data, and a target node set is randomly created for target worker nodes;
step S14: carrying out non-zero value updating on the adjacent matrix by using the target node set to obtain an updated adjacent matrix, and updating model parameters on each worker node in the target node set by using the updated adjacent matrix;
after initializing the model parameters of the target worker node and the adjacent matrix in the distributed heterogeneous system, controlling the target worker node to load preset target model parameters, randomly selecting sample data of kth iterative training for the target worker node, and when randomly selecting the sample data of the kth iterative training for the target worker node, performing gradient updating on the target model parameters by using the selected sample data and randomly creating a target node set for the target worker node.
In practical application, a target worker node worker can be defined i With the target model parameter of x i Wherein x is i Containing T parameters, all worker nodes in the distributed heterogeneous system can be spliced into a parameter matrix X = [ X ] 1 ,x 2 ,x 3 ,...,x n ]∈R T×N In the formula, N represents that the distributed heterogeneous system comprises N worker nodes. And, can define
A random sampling process of training samples for the target worker node, wherein k represents the kth iteration of the AI algorithm model, i represents an id address corresponding to the target worker node,
and showing that a set of sample data is randomly selected for the ith worker node in the kth iteration process. Sample data selected by a target worker node is assumed to be
Then, the process of gradient updating the target model parameters can be expressed as
Randomly creating a target node set g = { w } for a target worker node 1 ,w 2 ,...,w l And then, updating a non-zero value of the adjacency matrix by using the target node set to obtain an updated adjacency matrix, and updating the model parameters on each worker node in the target node set by using the updated adjacency matrix. Obviously, the purpose of this step is to make the target worker node know which model parameters on the worker node need to be updated.
Step S15: when the target worker node finishes the kth iterative training, judging whether an AI algorithm model on the target worker node converges;
step S16: if yes, the target worker node is judged to complete the kth AI algorithm model training task, the step of controlling the target worker node to load preset target model parameters is repeatedly executed, and the distributed heterogeneous system is judged to complete the AI acceleration task until the target worker node completes M times of iterative training;
wherein M is more than or equal to 2, and M is the preset iteration times.
And when the target worker node completes the kth iterative training, judging whether an AI algorithm model on the target worker node is converged, if so, indicating that the target worker node has completed the kth iterative training process, and in this case, returning to the step S12 again until the target worker node completes the M iterative training, indicating that the distributed heterogeneous system has completed an AI acceleration task.
It should be noted that, in this embodiment, the number M of times that the target worker node performs the iterative training is determined by the training precision of the AI algorithm model, that is, if the training precision required by the AI algorithm model is higher, the setting value of M is larger, and if the training precision required by the AI algorithm model is lower, the setting value of M is relatively reduced.
Compared with the prior art, in the embodiment, each iteration process only needs the parameter synchronization between the target worker node and the randomly selected worker node, and does not need the parameter synchronization with all the worker nodes, so that the requirement of each worker node in the distributed computing cluster on the communication bandwidth during the parameter synchronization can be reduced by the method provided by the embodiment. Moreover, the parameter updating method of the AI distributed training system provided by the embodiment is directed to a distributed heterogeneous system, and is not limited to a homogeneous distributed computing environment, so that the method provided by the embodiment can also improve the heterogeneity and the extensibility of the AI distributed training system.
In addition, the technical scheme provided by this embodiment starts from improving the heterogeneity and expandability of the distributed heterogeneous system, and randomly selects the Parameter synchronization node on the basis of the All-Reduce algorithm, so that the high expandability of the All-Reduce algorithm is ensured, and meanwhile, the defects that the overall performance of the AI distributed training system is limited by a short board of a worker node with slowest computational performance in a distributed computing environment and the communication pressure of All the worker nodes is too large are avoided by calculating a heterogeneous execution mode among the nodes, and the bandwidth bottleneck of the Parameter-Server algorithm is broken through, and the problem that the Ring-All-Reduce algorithm cannot fully exert the communication bandwidth performance due to too small Parameter data blocks is avoided.
As can be seen, in this embodiment, when the distributed heterogeneous system needs to complete an AI acceleration task, an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system, and model parameters and an adjacency matrix of a target worker node are initialized; then, controlling a target worker node to load preset target model parameters, and randomly selecting sample data of kth iterative training for the target worker node; and then, carrying out gradient updating on target model parameters based on sample data, randomly creating a target node set for target worker nodes, after the target node set is created, carrying out nonzero value updating on an adjacent matrix by using the target node set to obtain an updated adjacent matrix, updating model parameters on each worker node in the target node set by using the updated adjacent matrix, judging whether an AI algorithm model on the target worker nodes converges or not when the target worker nodes finish the kth iterative training, if the IA algorithm model on the target worker nodes converges, indicating that the target worker nodes finish the kth AI algorithm model training task, and finally, repeatedly executing the step of controlling the target worker nodes to load preset target model parameters until the target worker nodes finish M iterative training, and indicating that the distributed heterogeneous system finishes the AI acceleration task. Obviously, in the parameter synchronization method provided in this embodiment, because an AI acceleration task is performed based on a distributed heterogeneous system, and in the process of parameter synchronization by an AI model, a target worker node is a target node set created at random, and only a randomly selected worker node is subjected to parameter update, the method can enable an AI distributed training system to support a mixed heterogeneous distributed computing environment, and can also reduce the requirement on communication bandwidth when parameter synchronization is performed on each worker node in a distributed computing cluster.
Based on the foregoing embodiment, this embodiment further describes and optimizes the technical solution, and as a preferred implementation, the parameter updating method of the AI distributed training system further includes:
and constructing a distributed heterogeneous system by utilizing a plurality of worker nodes provided with the GPU and/or AI chips and/or FPGA.
It can be understood that, in practical applications, a GPU (Graphics Processing Unit), an AI (Artificial Intelligence) chip and an FPGA (Field Programmable Gate Array) are relatively common novel computing devices, and are also widely applied in actual life, and therefore, in this embodiment, a distributed heterogeneous system is built by using a plurality of worker nodes on which the GPUs and/or the AI chips and/or the FPGA are installed.
Obviously, the technical solution provided by this embodiment can make the parameter updating method of the AI distributed training system provided by this application more universal.
Based on the above embodiment, this embodiment further describes and optimizes the technical solution, and as a preferred implementation, the above steps: after the process of judging whether the AI algorithm model on the target worker node converges, the method further comprises the following steps:
and if not, re-executing the steps of controlling the target worker node to load preset target model parameters and randomly selecting sample data of the kth iterative training for the target worker node.
In this embodiment, when the target worker node completes the kth iterative training, if it is determined that the AI algorithm model on the target worker node is not converged, the step S12 needs to be executed again: and controlling the target worker node to load preset target model parameters, and randomly selecting sample data of the kth iterative training for the target worker node until the target worker node finishes the kth AI algorithm model training task.
Obviously, the integrity of the parameter updating method of the AI distributed training system provided by the present application can be further ensured by the technical solution provided by the present embodiment.
Based on the above embodiments, this embodiment further describes and optimizes the technical solution, and as a preferred implementation, the above steps: the process of initializing the model parameters and the adjacency matrix of the target worker node comprises the following steps:
and initializing the adjacent matrix of the target worker node to be a zero matrix E.
Specifically, in this embodiment, the adjacent matrix of the target worker node is initialized to the zero matrix E, and it can be thought that when the adjacent matrix of the target worker node is initialized to the zero matrix E, the subsequent parameter updating process can be greatly simplified, so that the resource overhead required by the AI distributed training system in the parameter updating process can be further reduced.
As a preferred embodiment, the process of updating the adjacency matrix with a non-zero value by using the target node set to obtain an updated adjacency matrix includes:
carrying out non-zero value updating on the zero matrix E by using the node set g to obtain an updated adjacent matrix;
wherein, the expression for updating the adjacency matrix is as follows:
in the formula, g is a node set, i and j are respectively the number of rows and columns in the zero matrix E, and u represents the number of rows and columns where i and j are the same.
In the present embodiment, a specific implementation is provided for performing a non-zero value update on the zero matrix E by using the node set g. In this embodiment, the distributed heterogeneous system shown in fig. 2 is specifically described, and it is assumed that a 3 rd worker node in the distributed heterogeneous system shown in fig. 2 is started and is denoted as w 3 Then, for worker node w 3 Proceed initialization, load w 3 The model parameter x set in advance 3 And is w 3 Randomly selecting sample data
Then based on the sample data
To w 3 Model parameter x of 3 A gradient update is performed, i.e.,
and is w 3 Creating a node set g = { w1, w3, w4}, and finally, performing nonzero value updating on a zero matrix E by using the node set g to obtain an updated adjacent matrix, wherein an expression of the updated adjacent matrix is as follows:
as a preferred embodiment, the above steps: the process of updating the model parameters on each worker node in the target node set by using the updated adjacency matrix comprises the following steps:
updating the model parameters on each worker node in the target node set by using the updated adjacency matrix to obtain updated parameters;
wherein, the expression of the update parameter is:
in the formula (I), the compound is shown in the specification,
a parameter matrix updated based on the node set g parameters in the kth iteration process, E k And g is an adjacency matrix updated based on the node set g in the k iteration process.
In this embodiment, a method for updating model parameters on each worker node in a target node set by using an updated adjacency matrix is provided, and in this embodiment, the 3 rd worker node in the distributed heterogeneous system shown in fig. 2 is also used as an example, when model parameters on each node in a node set g = { w1, w3, w4} are updated by using an updated adjacency matrix, actually, a process of accumulating and averaging model parameters of worker nodes w1, w3, and w4, that is, x is, a process of accumulating and averaging model parameters of worker nodes w1, w3, and w4 k+1 ←(x 1 +x 2 +x 4 )/3。
Based on the technical content disclosed in the above embodiments, in order to enable those skilled in the art to more clearly know the inventive principle of the present invention, the present embodiment specifically describes the parameter updating process of the target worker node through the following pseudo code.
Step1: to target worker node worker i Initializing the model parameters and the adjacency matrix E to control a target worker node worker i Loading model parameters x i ；
Step2: is worker node worker i Randomly selecting sample data of kth iterative training
Step3: target worker node worker based on gradient calculation i Is subjected to gradient update
Step4: randomly creating a target node set g = { w } for a target worker node 1 ,w 2 ,...,w l }；
Step5: g = { w) from the set of target nodes 1 ,w 2 ,...,w l The adjacency matrix E is updated, that is,
step6: updating model parameters on each worker node in the target node set by utilizing the updated adjacency matrix
Step7: and completing the parameter updating of the kth iteration.
Referring to fig. 3, fig. 3 is a structural diagram of a parameter updating apparatus of an AI distributed training system according to an embodiment of the present invention, where the parameter updating apparatus includes:
the model initialization module 21 is configured to start a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system when the distributed heterogeneous system needs to complete an AI acceleration task, and initialize model parameters and an adjacency matrix of the target worker node; an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system;
the sample selection module 22 is used for controlling the target worker node to load preset target model parameters and randomly selecting sample data of the kth iterative training for the target worker node; wherein k is more than or equal to 1;
the gradient updating module 23 is configured to perform gradient updating on the target model parameters based on the sample data, and randomly create a target node set for the target worker node;
the parameter updating module 24 is configured to perform non-zero value updating on the adjacent matrix by using the target node set to obtain an updated adjacent matrix, and update the model parameters on each worker node in the target node set by using the updated adjacent matrix;
the convergence judging module 25 is configured to, when the target worker node completes the kth iterative training, judge whether an AI algorithm model on the target worker node converges;
the task training module 26 is configured to, when the determination result of the convergence determination module is yes, determine that the target worker node completes a kth AI algorithm model training task, repeatedly execute the step of controlling the target worker node to load preset target model parameters, and determine that the distributed heterogeneous system completes an AI acceleration task until the target worker node completes M iterative trainings; wherein M is more than or equal to 2, and M is the preset iteration number.
The parameter updating device of the AI distributed training system provided by the embodiment of the invention has the beneficial effects of the parameter updating method of the AI distributed training system.
Referring to fig. 4, fig. 4 is a structural diagram of a parameter updating device of an AI distributed training system according to an embodiment of the present invention, where the parameter updating device includes:
a memory 31 for storing a computer program;
a processor 32, configured to implement a parameter updating method of the AI distributed training system as disclosed in the foregoing when executing the computer program.
The parameter updating device of the AI distributed training system provided by the embodiment of the invention has the beneficial effects of the parameter updating method of the AI distributed training system.
Accordingly, the present invention also discloses a computer readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for updating parameters of the AI distributed training system as disclosed above is implemented.
The computer-readable storage medium provided by the embodiment of the invention has the beneficial effects of the parameter updating method of the AI distributed training system disclosed in the foregoing.
In the present specification, the embodiments are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
The parameter updating method, device, equipment and medium of the AI distributed training system provided by the invention are introduced in detail, and a specific example is applied in the text to explain the principle and the implementation of the invention, and the description of the above embodiment is only used to help understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (6)
1. A parameter updating method of an AI distributed training system is characterized by comprising the following steps:
when the distributed heterogeneous system needs to finish an AI acceleration task, starting a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system, and initializing model parameters and an adjacent matrix of the target worker node; an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system;
controlling the target worker node to load preset target model parameters, and randomly selecting sample data of a kth iterative training for the target worker node; wherein k is more than or equal to 1;
gradient updating is carried out on the target model parameters based on the sample data, and a target node set is randomly created for the target worker nodes;
updating a non-zero value of the adjacent matrix by using the target node set to obtain an updated adjacent matrix, and updating model parameters on each worker node in the target node set by using the updated adjacent matrix;
when the target worker node completes the kth iterative training, judging whether an AI algorithm model on the target worker node converges;
if yes, the target worker node is judged to complete the kth AI algorithm model training task, the step of controlling the target worker node to load preset target model parameters is repeatedly executed, and the distributed heterogeneous system is judged to complete the AI acceleration task until the target worker node completes M times of iterative training; wherein M is more than or equal to 2, and M is a preset iteration number;
the process of initializing the model parameters and the adjacency matrix of the target worker node comprises the following steps:
initializing the adjacent matrix of the target worker node as a zero matrix E;
the process of updating the adjacency matrix with the non-zero value by using the target node set to obtain the updated adjacency matrix comprises the following steps:
carrying out non-zero value updating on the zero matrix E by utilizing a node set g to obtain an updated adjacent matrix;
wherein the expression of updating the adjacency matrix is as follows:
in the formula, g is the node set, i and j are the number of rows and columns in the zero matrix E respectively, and u represents the number of rows and columns with the same i and j;
the process of updating the model parameters on each worker node in the target node set by using the updated adjacency matrix comprises the following steps:
updating the model parameters on each worker node in the target node set by using the updated adjacency matrix to obtain updated parameters;
wherein the expression of the update parameter is:
in the formula (I), the compound is shown in the specification,
a parameter matrix updated based on the parameters of the node set g in the k iteration process,
and updating the adjacency matrix based on the node set g in the k iteration process.
2. The parameter updating method according to claim 1, further comprising:
and building the distributed heterogeneous system by using a plurality of worker nodes provided with the GPU and/or AI chips and/or the FPGA.
3. The method according to claim 1, wherein after the process of determining whether the AI algorithm model on the target worker node converges, the method further comprises:
and if not, re-executing the step of controlling the target worker node to load preset target model parameters and randomly selecting sample data of the kth iterative training for the target worker node.
4. A parameter updating apparatus of an AI distributed training system, comprising:
the model initialization module is used for starting a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system when the distributed heterogeneous system needs to finish an AI acceleration task, and initializing model parameters and an adjacent matrix of the target worker node; an AI algorithm model is deployed on all worker nodes of the distributed heterogeneous system;
the sample selection module is used for controlling the target worker node to load preset target model parameters and randomly selecting sample data of kth iterative training for the target worker node; wherein k is more than or equal to 1;
the gradient updating module is used for carrying out gradient updating on the target model parameters based on the sample data and randomly creating a target node set for the target worker node;
the parameter updating module is used for updating a nonzero value of the adjacency matrix by using the target node set to obtain an updated adjacency matrix and updating model parameters on each worker node in the target node set by using the updated adjacency matrix;
the convergence judging module is used for judging whether an AI algorithm model on the target worker node converges or not when the k-th iterative training is completed by the target worker node;
the task training module is used for judging that the target worker node completes the kth AI algorithm model training task when the judgment result of the convergence judgment module is yes, repeatedly executing the step of controlling the target worker node to load preset target model parameters until the target worker node completes M times of iterative training, and judging that the distributed heterogeneous system completes the AI acceleration task; wherein M is more than or equal to 2, and M is a preset iteration number;
the process of initializing the model parameters and the adjacency matrix of the target worker node comprises the following steps:
initializing the adjacent matrix of the target worker node to be a zero matrix E;
the process of updating the adjacency matrix with the non-zero value by using the target node set to obtain the updated adjacency matrix includes:
carrying out non-zero value updating on the zero matrix E by utilizing a node set g to obtain an updated adjacent matrix;
wherein the expression of updating the adjacency matrix is as follows:
in the formula, g is the node set, i and j are the number of rows and columns in the zero matrix E respectively, and u represents the number of rows and columns with the same i and j;
the process of updating the model parameters on each worker node in the target node set by using the updated adjacency matrix comprises the following steps:
updating model parameters on each worker node in the target node set by using the updated adjacency matrix to obtain updated parameters;
wherein the expression of the update parameter is:
in the formula (I), the compound is shown in the specification,
a parameter matrix updated based on the parameters of the node set g in the k iteration process,
and updating the adjacency matrix based on the node set g in the k iteration process.
5. A parameter updating apparatus of an AI distributed training system, comprising:
a memory for storing a computer program;
a processor for implementing a parameter updating method of an AI distributed training system as claimed in any one of claims 1 to 3 when executing the computer program.
6. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a processor, implements a parameter updating method of an AI distributed training system according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010820131.XA CN112001501B (en) | 2020-08-14 | 2020-08-14 | Parameter updating method, device and equipment of AI distributed training system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010820131.XA CN112001501B (en) | 2020-08-14 | 2020-08-14 | Parameter updating method, device and equipment of AI distributed training system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112001501A CN112001501A (en) | 2020-11-27 |
| CN112001501B true CN112001501B (en) | 2022-12-23 |
Family
ID=73473503
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010820131.XA Active CN112001501B (en) | 2020-08-14 | 2020-08-14 | Parameter updating method, device and equipment of AI distributed training system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112001501B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112766498B (en) * | 2021-01-29 | 2022-11-22 | 北京达佳互联信息技术有限公司 | Model training method and device |
| CN112965888B (en) * | 2021-03-03 | 2023-01-24 | 山东英信计算机技术有限公司 | Method, system, device and medium for predicting task quantity based on deep learning |
| CN113128700A (en) * | 2021-03-23 | 2021-07-16 | 同盾控股有限公司 | Method and system for accelerating safe multi-party computing federal model training |
| CN113485805B (en) * | 2021-07-01 | 2024-02-06 | 中科曙光(南京)计算技术有限公司 | Distributed computing adjustment method, device and equipment based on heterogeneous acceleration platform |
| CN113656494A (en) * | 2021-07-27 | 2021-11-16 | 中南大学 | Synchronization method and system of parameter server and readable storage medium |
| CN115250253B (en) * | 2022-06-22 | 2024-02-27 | 西南交通大学 | Reduction processing method for bandwidth perception and training method for AI model |
| CN115879543B (en) * | 2023-03-03 | 2023-05-05 | 浪潮电子信息产业股份有限公司 | Model training method, device, equipment, medium and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109492753A (en) * | 2018-11-05 | 2019-03-19 | 中山大学 | A kind of method of the stochastic gradient descent of decentralization |
| US20190318268A1 (en) * | 2018-04-13 | 2019-10-17 | International Business Machines Corporation | Distributed machine learning at edge nodes |
| CN111027708A (en) * | 2019-11-29 | 2020-04-17 | 杭州电子科技大学舟山同博海洋电子信息研究院有限公司 | Distributed machine learning-oriented parameter communication optimization method |
| CN111353582A (en) * | 2020-02-19 | 2020-06-30 | 四川大学 | Particle swarm algorithm-based distributed deep learning parameter updating method |
-
2020
- 2020-08-14 CN CN202010820131.XA patent/CN112001501B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190318268A1 (en) * | 2018-04-13 | 2019-10-17 | International Business Machines Corporation | Distributed machine learning at edge nodes |
| CN109492753A (en) * | 2018-11-05 | 2019-03-19 | 中山大学 | A kind of method of the stochastic gradient descent of decentralization |
| CN111027708A (en) * | 2019-11-29 | 2020-04-17 | 杭州电子科技大学舟山同博海洋电子信息研究院有限公司 | Distributed machine learning-oriented parameter communication optimization method |
| CN111353582A (en) * | 2020-02-19 | 2020-06-30 | 四川大学 | Particle swarm algorithm-based distributed deep learning parameter updating method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112001501A (en) | 2020-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112001501B (en) | Parameter updating method, device and equipment of AI distributed training system | |
| CN112101530B (en) | Neural network training method, device, equipment and storage medium | |
| CN110135584B (en) | Large-scale symbolic regression method and system based on adaptive parallel genetic algorithm | |
| CN108875955B (en) | Gradient lifting decision tree implementation method based on parameter server and related equipment | |
| US5524178A (en) | Neural network learning system | |
| CN111406264A (en) | Neural architecture search | |
| CN111626489B (en) | Shortest path planning method and device based on time sequence difference learning algorithm | |
| JP2021532437A (en) | Improving machine learning models to improve locality | |
| US20220050943A1 (en) | Learning device, inference device, and learned model | |
| US20160125103A1 (en) | Accelerating particle-swarm algorithms | |
| CN113311829A (en) | Multi-robot path planning method based on dynamic time window conflict search | |
| WO2020237798A1 (en) | Upgrade method and device | |
| CN111966122B (en) | Simulation control method and system for satellite and rocket separation | |
| CN113656891A (en) | Liquid rocket dynamic characteristic modeling analysis method and terminal equipment | |
| CN112306452A (en) | Method, device and system for processing service data by merging and sorting algorithm | |
| CN115437372B (en) | Robot path planning method and device, electronic equipment and storage medium | |
| CN112329941B (en) | Deep learning model updating method and device | |
| CN114001976A (en) | Method, device and equipment for determining control error and storage medium | |
| CN111151006B (en) | Method and device for generating collapsible building system in game and electronic equipment | |
| US11461665B2 (en) | Systems and methods of a Boolean network development environment | |
| WO2020105161A1 (en) | Edge device machine learning model switching system, edge device machine learning model switching method, program, and edge device | |
| CN113503885A (en) | Robot path navigation method and system based on sampling optimization DDPG algorithm | |
| TWI767122B (en) | Model constructing method, system, and non-transitory computer readable storage medium | |
| US9887904B2 (en) | Information processing apparatus, control method, and program | |
| WO2006033967A2 (en) | A method and apparatus for modeling systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |