WO2019128475A1 - 数据训练方法及装置、存储介质、电子装置 - Google Patents
数据训练方法及装置、存储介质、电子装置 Download PDFInfo
- Publication number
- WO2019128475A1 WO2019128475A1 PCT/CN2018/114209 CN2018114209W WO2019128475A1 WO 2019128475 A1 WO2019128475 A1 WO 2019128475A1 CN 2018114209 W CN2018114209 W CN 2018114209W WO 2019128475 A1 WO2019128475 A1 WO 2019128475A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- operator
- sub
- models
- split
- splitting
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
Definitions
- the present invention relates to the field of artificial intelligence, and in particular to a data training method and apparatus, a storage medium, and an electronic device.
- the training of Deep Learning Models requires a large amount of computational power, and the time required to complete a training often reaches several days or even months. Therefore, in order to speed up the training of the deep learning model, it is usually implemented by adding processing equipment and optimizing the training model, but the first type will increase the input of network resources, and the second type is difficult to implement in a short time.
- the embodiment of the invention provides a data training method and device, a storage medium and an electronic device.
- a data training method including: determining sample data and available cluster resources; splitting a total training model into a plurality of sub-models; and using the cluster resources on the plurality of sub-models
- the sample data is trained in parallel.
- a data training apparatus comprising: a determining module configured to determine sample data and available cluster resources; and a splitting module configured to split the total training model into a plurality of sub-models; A training module configured to use the cluster resources to train the sample data in parallel on the plurality of sub-models.
- a storage medium having stored therein a computer program, wherein the computer program is configured to execute the steps of any one of the method embodiments described above.
- an electronic device comprising a memory and a processor, wherein the memory stores a computer program, the processor being arranged to run the computer program to perform any of the above The steps in the method embodiments.
- FIG. 1 is a flow chart of a data training method according to an embodiment of the present invention.
- FIG. 2 is a block diagram showing the structure of a data training apparatus according to an embodiment of the present invention.
- FIG. 5 is a schematic diagram of a Split-Concat operator optimization scheme provided by this embodiment.
- FIG. 6 is a schematic diagram of a parameter-based splitting scheme provided by this embodiment.
- FIG. 7 is an interaction flowchart provided according to this embodiment.
- FIG. 1 is a flowchart of a data training method according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps:
- Step S102 determining sample data and available cluster resources
- Step S104 splitting the total training model into multiple sub-models
- Step S106 using the cluster resource to train the sample data in parallel on the plurality of sub-models.
- the execution body of the foregoing steps may be a server, a data processing system, a cluster platform, etc., but is not limited thereto, and may be applied in a scenario of a deep learning model and a neural network model.
- splitting the total training model into a plurality of sub-models includes at least one of: splitting the total training model into a plurality of first sub-models, wherein the plurality of first sub-models are connected in parallel; The model is split into a plurality of second sub-models, wherein a plurality of second sub-models are connected in series.
- splitting the total training model into a plurality of first sub-models comprises one of: splitting the total training model into a plurality of first sub-models according to the indication information; the indication information may be user-entered or System generated.
- the total training model is split into a plurality of first sub-models according to the type of the operator, wherein the total training model is composed of one or more operators.
- splitting the overall training model into a plurality of first sub-models according to the indication information includes:
- the total training model is split into N first sub-models whose input is (B/N) ⁇ I, where B is the first batch of dimensions, and the size of the first batch is the same as the size of the batch size, and I is The dimension of the input vector of the total training model, N is an integer greater than one.
- the first submodel includes a sub-dense operator.
- splitting the overall training model into a plurality of first sub-models according to the type of operator includes:
- the dense operator is divided into N sub-diffuse operators whose calculation parameters are (I/N) ⁇ O, wherein the sub-dense operator is the same as the input tensor (Tensor) of the dense operator, and O is a dense calculation.
- the dimension of the output vector of the child I is the dimension of the input vector of the dense operator, N is an integer greater than 1;
- the convolution operator is split into N sub-convolution operators, where the input of the sub-convolution operator
- the input tensor of the volume convolution operator is the same, and one sub-dense operator includes a plurality of calculation parameters; wherein the first sub-model includes at least one of the following: a sub-dense operator, a sub-convolution operator.
- splitting the overall training model into a plurality of second sub-models includes:
- determining the sample data and the available cluster resources includes: receiving a training job, acquiring corresponding sample data from the training job; determining a first processor that is currently idle in the cluster, and receiving the specified second processor information, The available processor resources are determined in the first processor based on the second processor information, wherein the cluster resources include processor resources.
- the processor can be a CPU, a GPU, an MPU, etc.
- using the cluster resource to train the sample data in parallel on the plurality of sub-models includes: dividing the sample data into M fragments, and inputting to the M*K sub-models of the cluster resources in parallel for training, wherein K is The minimum cluster resource required to configure a submodel, M is an integer greater than 0, and K is an integer greater than zero.
- K is The minimum cluster resource required to configure a submodel
- M is an integer greater than
- K is an integer greater than zero.
- M and K the following three parallel ways can be realized: data parallel, model parallel, mixed parallel, when M is greater than 1, data parallel, input M different slices into the sub-model in parallel, when K is greater than 1,
- the model is parallel, using the K submodel to process one slice in parallel, and the hybrid parallel is the superposition of the two.
- the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
- the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
- the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
- a data training device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has not been described again.
- the term “module” may implement a combination of software and/or hardware of a predetermined function.
- the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
- FIG. 2 is a structural block diagram of a data training apparatus according to an embodiment of the present invention. As shown in FIG. 2, the apparatus includes:
- a determining module 20 configured to determine sample data and available cluster resources
- the splitting module 22 is configured to split the total training model into multiple sub-models
- the training module 24 is configured to use the cluster resources to train the sample data in parallel on the plurality of sub-models.
- the splitting module includes at least one of the following: a first splitting unit configured to split the total training model into a plurality of first sub-models, wherein the plurality of first sub-models are connected in parallel; the second splitting And a unit configured to split the total training model into a plurality of second sub-models, wherein the plurality of second sub-models are serially connected.
- each of the above modules may be implemented by software or hardware.
- the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the above modules are in any combination.
- the forms are located in different processors.
- This embodiment is an optional embodiment of the present application, which is used to explain and explain the application in detail in combination with a specific model example:
- a parallel computing method can be used, that is, a training is divided into multiple sub-parts, and each sub-part is simultaneously calculated on different computing devices, thereby achieving the purpose of speeding up training.
- deep learning parallel computing including parallelism of data parallelism and model parallelism, it is necessary to select appropriate parallel algorithms according to the characteristics of the model and computing cluster.
- This embodiment is a method and system capable of selecting an appropriate parallel algorithm according to the characteristics of the deep learning model and the characteristics of the high performance cluster, and automatically transforming the original deep learning model so that it can generate a larger Calculate the degree of parallelism for faster training. It is a method used to enable the deep learning model to automatically perform parallel training on a high performance computing cluster.
- the problem to be solved in this embodiment is to implement an automated parallel training deep learning model: the user only needs to specify the number of nodes used by the training model (in this embodiment, the GPU is taken as an example), and the model to be trained (for example, DNN (deep nerve) Network), CNN (convolutional neural network) or RNN (circular neural network), etc., the system automatically selects the parallel training algorithm and transforms the model accordingly to maximize the parallelism of the algorithm, thus achieving efficient parallelism. training.
- DNN deep nerve
- CNN convolutional neural network
- RNN circular neural network
- the embodiment provides a system for implementing an automated training deep learning model, including: an application manager, a resource manager, a job scheduler, and an executable.
- an application manager a resource manager, a job scheduler, and an executable.
- the function of each module in the method is described in detail below:
- Application Manager A service process running on a High Performance Computing (HPC) cluster. Responsible for the management of training operations, including the start and stop of the job, the main control of the work of other modules.
- HPC High Performance Computing
- Resource Manager A service process running on an HPC cluster. Responsible for deciding which algorithm to use to train the deep learning model submitted by the user and allocate the corresponding resources on the HPC cluster. This process includes the following algorithms and processes:
- Step A Obtain a memory size M available to the node (GPU) on the HPC cluster;
- Step B Obtain the number of nodes D designated by the user
- Step C Calculate the memory size R that the deep learning model needs to use by traversing all the operators of the deep learning model, and finding the size of the output Tensor of all the operators; plus the size of all the parameters in the model. which is:
- size(out(i)) is the output size of the operator i
- size(j) is the size of the parameter j
- S is the data type size. For example, for float32, its size is 4. It is an additional memory factor for video memory. The actual memory required by different frameworks is larger than the calculated memory. The default value is 1.1.
- Step D Determine the distribution granularity G.
- the granularity of allocation is the minimum number of allocated GPUs needed to drop a model. In order to reduce fragmentation, we limit the allocation granularity to be an integer power of 2, namely 1, 2, 4, 8... Therefore, the final distribution granularity is:
- N min ⁇ n
- Step E Determine the data parallelism DP.
- A will be equal to the number of nodes specified by the user.
- FIG. 3 is a schematic diagram of a parallel algorithm provided by this embodiment.
- Job scheduler Each training task contains a job scheduler process, which is responsible for transforming the deep learning model to improve its parallelism, and then assigning the split model to multiple actuators for distributed parallel training.
- the method of increasing the degree of parallelism of deep learning is to split based on the operator, that is, by splitting an operator into multiple operators, each split operator can realize parallel computing on different nodes, thereby increasing The degree of concurrency of the calculation.
- split There are two ways to split: input-based splits and parameter-based splits.
- the input-based splitting method is:
- the user specifies the batch size, that is, how many training samples are input in one iteration.
- the training sample adds a batch dimension whose size is batch size. For example, if the input of the original operator is an I-dimensional vector, the output is an O-dimensional vector, and the batch size is B, the input can be expressed as a B ⁇ I Tensor.
- an operator that receives a Tensor input of B ⁇ I it can be split into N operators that receive Tensor with input (B/N) ⁇ I.
- the Split operator is responsible for splitting the Tensor whose original input is B ⁇ I into N (B/N) ⁇ I Tensor as the input of the new N operators; the Concat operator is responsible for the N (B/N)
- the output of Ten's Tensor is recombined into a B ⁇ O Tensor. In this way, we guarantee that the split operator is equivalent to the operator before splitting.
- FIG. 5 is a schematic diagram of a Split-Concat operator optimization scheme provided by this embodiment.
- OP is an operator, including: for an adjacent Concat-Split operator pair, if the Concat operator inputs the dimensions and size of the Tensor, The Concat-Split operator can be optimized by the same size and size as the Output Tensor of the Split operator, and the Concat's indegree is equal to the Split's degree.
- FIG. 6 is a schematic diagram of a parameter-based splitting scheme provided by this embodiment, where parameter-based splitting includes: implementing different splitting schemes according to types of operators.
- the Dense operator if the size of the input Tensor is B ⁇ I and the size of the parameter Tensor is I ⁇ O, the output Tensor size is B ⁇ O.
- the original Dense operator is split into N Dense operators.
- the input of each Dense operator is still B ⁇ I, and the parameter Tensor is split into I ⁇ (O/N), then the Tensor size of each operator output. It is B ⁇ (O / N).
- the Spilt operator is responsible for splitting the parameter Tensor of size I ⁇ O into N parameters Tensor of size I ⁇ (O/N); the Concat operator is responsible for recombining the output of Ten B ⁇ (O/N) Tensor It is a Tensor of size B ⁇ O. In this way, we guarantee that the split operator is equivalent to the operator before splitting.
- Each worker node contains an executor process responsible for performing the training assigned to the (partial) deep learning model on this node.
- the executor is divided into two types, the Worker and the Parameter Server, which complete the training of the parameters of the model and the summary of the parameters.
- FIG. 7 is an interaction flowchart provided according to this embodiment, including:
- Step A The user submits a training job to the application manager, specifies a deep learning model to be trained, and the number of nodes that he wishes to use, and the application manager sends it to the resource manager;
- Step B The resource manager calculates the allocation granularity G and the data parallelism DP, and determines a parallel algorithm (data parallel, model parallel or hybrid parallel) by G and DP; assigns idle nodes to the training job on the HPC according to G and DP;
- Step C The application manager starts the job scheduler, and passes the model submitted by the user, the resource manager allocates the resource and its parameters;
- Step D The job scheduler splits the model into G sub-models according to the input granularity or the parameter-based splitting method according to the allocation granularity G, and performs Split-Concat operator optimization on the G sub-models;
- Step E The job scheduler starts DP*G executors, and each G executor forms an execution group, which is trained in a model parallel manner; the data is divided into DP shards, and the DP execution groups are trained in data parallel mode. ;
- Step F After all the actuators are executed, the application manager obtains the final trained model, deletes the training job, and releases the resources.
- the user can specify the number of GPUs to be used, and according to the characteristics of the deep learning model and the characteristics of the high-performance cluster, automatically generate a corresponding and efficient parallel computing solution. It can save the investment in algorithm research and development, and can also train the model faster.
- Embodiments of the present invention also provide a storage medium having stored therein a computer program, wherein the computer program is configured to perform the steps of any of the above method embodiments when executed.
- the storage medium described above can be configured to store a computer program for performing the following steps:
- the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, and a magnetic memory.
- ROM Read-Only Memory
- RAM Random Access Memory
- mobile hard disk a magnetic memory.
- magnetic memory A variety of media that can store computer programs, such as a disc or an optical disc.
- Embodiments of the present invention also provide an electronic device comprising a memory and a processor having a computer program stored therein, the processor being arranged to execute a computer program to perform the steps of any of the method embodiments described above.
- the electronic device may further include a transmission device and an input and output device, wherein the transmission device is connected to the processor, and the input and output device is connected to the processor.
- the memory can be either volatile memory or non-volatile memory, and can include both volatile and nonvolatile memory.
- the non-volatile memory may be a ROM, a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), or an electrically erasable device.
- EEPROM Electrically Erasable Programmable Read-Only Memory
- FRAM Ferromagnetic random access memory
- Flash Memory Magnetic Surface Memory, Optical Disk, or Read Only Disc (CD) -ROM, Compact Disc Read-Only Memory
- the magnetic surface memory may be a disk storage or a tape storage.
- the volatile memory can be a random access memory (RAM) that acts as an external cache.
- RAM Static Random Access Memory
- SSRAM Synchronous Static Random Access Memory
- SSRAM Dynamic Random Access
- DRAM Dynamic Random Access Memory
- SDRAM Synchronous Dynamic Random Access Memory
- DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
- ESDRAM enhancement Enhanced Synchronous Dynamic Random Access Memory
- SLDRAM Synchronous Dynamic Random Access Memory
- DRRAM Direct Memory Bus Random Access Memory
- the method disclosed in the foregoing embodiments of the present invention may be applied to a processor or implemented by a processor.
- the processor may be an integrated circuit chip with signal processing capabilities.
- each step of the above method may be completed by an integrated logic circuit of hardware in a processor or an instruction in a form of software.
- the above processor may be a general purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or the like.
- DSP digital signal processor
- the processor may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present invention.
- a general purpose processor can be a microprocessor or any conventional processor or the like.
- the steps of the method disclosed in the embodiment of the present invention may be directly implemented as a hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor.
- the software module can be located in a storage medium, the storage medium being located in the memory, the processor reading the information in the memory, and completing the steps of the foregoing methods in combination with the hardware thereof.
- the electronic device may be configured by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), and Complex Programmable Logic Devices (CPLDs).
- ASICs Application Specific Integrated Circuits
- DSPs Digital Signal processors
- PLDs Programmable Logic Devices
- CPLDs Complex Programmable Logic Devices
- Complex Programmable Logic Device Field-Programmable Gate Array (FPGA), General Purpose Processor, Controller, Micro Controller Unit (MCU), Microprocessor, or other electronic components Implemented to perform the aforementioned method.
- FPGA Field-Programmable Gate Array
- MCU Micro Controller Unit
- Microprocessor or other electronic components Implemented to perform the aforementioned method.
- the processor described above can be configured to perform the following steps by a computer program:
- modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
- the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
- the invention is not limited to any specific combination of hardware and software.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Complex Calculations (AREA)
- Machine Translation (AREA)
Abstract
本发明提供了一种数据训练方法及装置、存储介质、电子装置,其中,该方法包括:确定样本数据和可用的集群资源;将总训练模型拆分为多个子模型;使用集群资源在多个子模型上并行训练样本数据。通过本发明,解决了相关技术中训练样本数据效率过低的技术问题。
Description
相关申请的交叉引用
本申请基于申请号为201711488171.3、申请日为2017年12月29日的中国专利申请提出,并要求该中国专利申请的优先权,该中国专利申请的全部内容在此引入本申请作为参考。
本发明涉及人工智能领域,具体而言,涉及一种数据训练方法及装置、存储介质、电子装置。
相关技术中,深度学习模型(Deep Learning Models)的训练需要消耗庞大的计算力,完成一次训练的耗时经常达到数天甚至数月。因此,为了加快深度学习模型的训练,通常采用增加处理设备,优化训练模型来实现,但第一种会加大网络资源的投入量,第二种在短时间内也很难实现。
针对相关技术中存在的上述问题,目前尚未发现有效的解决方案。
发明内容
本发明实施例提供了一种数据训练方法及装置、存储介质、电子装置。
根据本发明的一个实施例,提供了一种数据训练方法,包括:确定样本数据和可用的集群资源;将总训练模型拆分为多个子模型;使用所述集群资源在所述多个子模型上并行训练所述样本数据。
根据本发明的另一个实施例,提供了一种数据训练装置,包括:确定模块,配置为确定样本数据和可用的集群资源;拆分模块,配置为将总训 练模型拆分为多个子模型;训练模块,配置为使用所述集群资源在所述多个子模型上并行训练所述样本数据。
根据本发明的又一个实施例,还提供了一种存储介质,所述存储介质中存储有计算机程序,其中,所述计算机程序被设置为运行时执行上述任一项方法实施例中的步骤。
根据本发明的又一个实施例,还提供了一种电子装置,包括存储器和处理器,所述存储器中存储有计算机程序,所述处理器被设置为运行所述计算机程序以执行上述任一项方法实施例中的步骤。
通过本发明,通过将总训练模型拆分为多个子模型,然后在多个子模型上并行训练样本数据,解决了相关技术中训练样本数据效率过低的技术问题,在不增加网络资源的情况下,提高了训练样本数据的速度。
此处所说明的附图用来提供对本发明的进一步理解,构成本申请的一部分,本发明的示意性实施例及其说明用于解释本发明,并不构成对本发明的不当限定。在附图中:
图1是根据本发明实施例的数据训练方法的流程图;
图2是根据本发明实施例的数据训练装置的结构框图;
图3是本实施例提供的并行算法示意图;
图4是本实施例提供的基于输入拆分方案示意图;
图5是本实施例提供的Split-Concat算子优化方案示意图;
图6是本实施例提供的基于参数拆分方案示意图;
图7是根据本实施例提供的交互流程图。
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是, 在不冲突的情况下,本申请中的实施例及实施例中的特征可以相互组合。
需要说明的是,本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象,而不必用于描述特定的顺序或先后次序。
在本实施例中提供了一种数据训练方法,图1是根据本发明实施例的数据训练方法的流程图,如图1所示,该流程包括如下步骤:
步骤S102,确定样本数据和可用的集群资源;
步骤S104,将总训练模型拆分为多个子模型;
步骤S106,使用集群资源在多个子模型上并行训练样本数据。
通过上述步骤,通过将总训练模型拆分为多个子模型,然后在多个子模型上并行训练样本数据,解决了相关技术中训练样本数据效率过低的技术问题,在不增加网络资源的情况下,提高了训练样本数据的速度。
在一些实施例中,上述步骤的执行主体可以为服务器,数据处理***,集群平台等,但不限于此,可以应用在深度学习模型和神经网络模型的场景中。
在一些实施例中,将总训练模型拆分为多个子模型包括以下至少之一:将总训练模型拆分为多个第一子模型,其中,多个第一子模型并行连接;将总训练模型拆分为多个第二子模型,其中,多个第二子模型串行连接。
在一些实施例中,将总训练模型拆分为多个第一子模型包括以下之一:根据指示信息将总训练模型拆分为多个第一子模型;指示信息可以是用户输入的或者是***生成的。根据算子的类型将总训练模型拆分为多个第一子模型,其中,总训练模型由一个或多个算子组成。
在一个示例中,根据指示信息将总训练模型拆分为多个第一子模型包括:
S11,获取指示信息,其中,指示信息用于指示总训练模型的批尺寸 (batch size);批尺寸用于描述一次迭代输入多少个训练样本;
S12,将总训练模型拆分为N个输入为(B/N)×I的第一子模型,其中,B为第一批维度,第一批维度的大小与批尺寸的大小相同,I为总训练模型的输入向量的维度,N为大于1的整数。第一子模型包括子稠密算子。
在另一个示例中,根据算子的类型将总训练模型拆分为多个第一子模型包括:
S21,获取算子的类型,其中,算子的类型包括:稠密(Dense)算子,卷积(Conv)算子;
S22,将稠密算子拆分为N个计算参数为(I/N)×O的子稠密算子,其中,子稠密算子与稠密算子的输入张量(Tensor)相同,O为稠密算子的输出向量的维度,I为稠密算子的输入向量的维度,N为大于1的整数;将卷积算子拆分为N个子卷积算子,其中,子卷积算子的输入张量为卷积算子的输入张量相同,一个子稠密算子包括多个计算参数;其中,第一子模型包括以下至少之一:子稠密算子,子卷积算子。
在一些实施例中,将总训练模型拆分为多个第二子模型包括:
S31,解析总训练模型得到多个算子,其中,总训练模型包括Concat算子和Split算子,串行相邻的Concat算子和Split算子组成第一Concat-Split算子对;
S32,在第一Concat-Split算子对中,在Concat算子的输入张量与Split算子的输出张量相同时,在总训练模型中删除第一Concat-Split算子对中的Concat算子和Split算子后,将总训练模型拆分为多个第二子模型。
在本实施例中,确定样本数据和可用的集群资源包括:接收训练作业,从训练作业中获取对应的样本数据;确定集群当前空闲的第一处理器,以及接收指定的第二处理器信息,根据第二处理器信息在第一处理器中确定可用的处理器资源,其中,集群资源包括处理器资源。处理器可以是CPU, GPU,MPU等,
在本实施例中,使用集群资源在多个子模型上并行训练样本数据包括:将样本数据划分为M个分片后,并行输入到集群资源的M*K个子模型上进行训练,其中,K为配置一个子模型所需要的最小集群资源,M为大于0的整数,K为大于0的整数。根据M和K的取值可以实现以下三种并行方式:数据并行、模型并行,混合并行,M大于1时,数据并行,将M个不同的分片并行输入到子模型,K大于1时,模型并行,使用K子模型并行处理一个分片,混合并行则是两者叠加。
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,或者网络设备等)执行本发明各个实施例所述的方法。
在本实施例中还提供了一种数据训练装置,该装置用于实现上述实施例及优选实施方式,已经进行过说明的不再赘述。如以下所使用的,术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现,但是硬件,或者软件和硬件的组合的实现也是可能并被构想的。
图2是根据本发明实施例的数据训练装置的结构框图,如图2所示,该装置包括:
确定模块20,配置为确定样本数据和可用的集群资源;
拆分模块22,配置为将总训练模型拆分为多个子模型;
训练模块24,配置为使用集群资源在多个子模型上并行训练样本数据。
可选的,拆分模块包括以下至少之一:第一拆分单元,配置为将总训练模型拆分为多个第一子模型,其中,多个第一子模型并行连接;第二拆分单元,配置为将总训练模型拆分为多个第二子模型,其中,多个第二子模型串行连接。
需要说明的是,上述各个模块是可以通过软件或硬件来实现的,对于后者,可以通过以下方式实现,但不限于此:上述模块均位于同一处理器中;或者,上述各个模块以任意组合的形式分别位于不同的处理器中。
本实施例是本申请的可选实施例,用于结合具体的模型实例对申请进行详细解释和说明:
为了加快深度学习模型的训练,可以使用并行计算的方法,即将一个训练分割为多个子部分,让每个子部分在不同的计算设备上同时完成计算,从而达到加快训练的目的。在深度学习并行计算中,包括数据并行和模型并行两大类并行算法,需要根据模型和计算集群的特征选择合适的并行算法。
本实施例是这样一种方法和***:能够根据深度学习模型的特征、以及高性能集群的特征,选择合适的并行算法,并且自动对原始的深度学习模型进行变换,使之能够产生更大的计算并行度,达到更快的训练速度的目的。是一种用来实现让深度学习模型在高性能计算集群上自动完成并行训练的方法。
本实施例所要解决的问题是实现自动化并行训练深度学习模型:用户只需要指定训练模型使用的节点(本实施例以GPU为例进行说明)个数,以及要训练的模型(例如DNN(深度神经网络)、CNN(卷积神经网络)或RNN(循环神经网络)等),***则自动选择并行训练算法,并对模型进行相应的变换,以尽可能提升算法的并行度,从而实现高效的并行训练。
本实施例提供一种实现自动化训练深度学习模型的***,包括:应用管理器(Application Manager),资源管理器(Resource Manager)、作业调度器(Job Scheduler)和执行器(Executor)四个模块,下边对所述方法中每个模块的功能进行详细描述:
应用管理器:运行在高性能计算(High Performance Computing,简称为HPC)集群上的一个服务进程。负责实现训练作业的管理,包括作业的启动、停止,主控其它各个模块的工作。
资源管理器:运行在HPC集群上的一个服务进程。负责决定采用何种算法训练用户提交的深度学习模型,并在HPC集群上分配相应的资源。这个过程包括以下算法和流程:
步骤A.获取HPC集群上,节点(GPU)可用的显存大小M;
步骤B.获取用户指定使用的节点个数D;
步骤C.计算深度学习模型需要使用的显存大小R,其方法为:遍历深度学习模型的所有算子,求出所有算子的输出Tensor的大小;再加上模型中所有参数的大小。即:
在上式中,size(out(i))是算子i的输出大小,size(j)是参数j的大小,S是数据类型大小。例如,对于float32来说,其大小为4。是显存额外显存因子,不同的框架实际需要的显存要比计算的显存数大一些,默认取值为1.1。
步骤D.确定分配粒度G。分配粒度是为了放下一个模型所需要的最小分配的GPU个数。为了减少碎片产生,我们限制分配粒度必须是2的整数次幂,即1、2、4、8……。因此,最终的分配粒度为:
G=2
N
N=min{n|2
n>ceil(R/M)}
步骤E.确定数据并行度DP。数据并行度表示整个训练数据分为多少个分片,其计算公式为:DP=floor(D/G)。
步骤F.根据G和DP,在HPC上分配资源节点(GPU)总数为A为:A=DP*G。
限制用户指定的DP只能为2的整数次幂,那么A将会等于用户指定的节点数DP。
根据不同的DP和G,我们可以将并行训练算法分为数据并行、模型并行和混合并行。其方法为:多个节点组成一个复制组,复制组内的节点采用模型并行来进行训练,G定义了复制组内包含的节点数。一个训练作业则包含了多个复制组,复制组之间采用数据并行来进行训练,DP定义了一个训练作业包含的复制组数。图3是本实施例提供的并行算法示意图。
作业调度器:每个训练任务包含一个作业调度器进程,负责对深度学习模型进行变换以提升其并行度,然后将拆分后的模型指派给多个执行器,实现分布式并行训练。
增加深度学习并行度的方法是基于算子进行拆分,即通过将一个算子拆分为多个算子,使每个拆分后的算子能够在不同的节点上实现并行计算,从而增加了计算的并发度。拆分有两种方法:基于输入的拆分和基于参数的拆分。
图4是本实施例提供的基于输入拆分方案示意图,其中,Size为尺寸,Axis为坐标轴,Conv2D 1,Conv2D 2,Conv2D 3为示意的第1二维卷积,第2二维卷积,第3二维卷积,基于输入的拆分方法为:每个深度学习的迭代(step)中,用户会指定batch size,即一次迭代输入多少个训练样本。在向神经网络进行输入时,训练样本会增加一个batch维度,其大小为batch size。例如,如果原始算子的输入为I维向量,输出为O维向量,batch size 为B,则输入可以表达为一个B×I的Tensor。对于一个接收输入为B×I的Tensor的算子,可以将其拆分为N个接收输入为(B/N)×I的Tensor的算子。为了实现拆分之后的算子和原算子等价,还需要增加Split算子和Concat算子。Split算子负责将原输入为B×I的Tensor拆分为N个(B/N)×I的Tensor,作为新的N个算子的输入;Concat算子负责将N个(B/N)×O的输出Tensor重新合并为一个B×O的Tensor。这样,我们就保证了拆分后的算子与拆分前的算子等价。
在基于输入的拆分方法之上,还可以优化以减少不必要的Split-Concat算子。图5是本实施例提供的Split-Concat算子优化方案示意图,OP(Operation)为算子,包括:对于相邻的Concat-Split算子对,如果Concat算子的输入Tensor的维度和大小,与Split算子的输出Tensor的维度和大小相等,且Concat的入度等于Split的出度,则这对Concat-Split算子都可以优化掉。
图6是本实施例提供的基于参数拆分方案示意图,基于参数的拆分包括:根据算子的类型实现不同的拆分方案。对于Dense算子,如果其输入的Tensor的大小为B×I,参数的Tensor大小为I×O,则输出的Tensor大小为B×O。将原始Dense算子拆分为N个Dense算子,每个Dense算子的输入仍然为B×I,参数的Tensor拆分为I×(O/N),则每个算子输出的Tensor大小为B×(O/N)。为了实现拆分之后的算子和原算子等价,还需要增加Split算子和Concat算子。Spilt算子负责将大小为I×O的参数Tensor拆分为N个大小为I×(O/N)的参数Tensor;Concat算子负责将N个B×(O/N)的输出Tensor重新合并为一个大小为B×O的Tensor。这样,我们就保证了拆分后的算子与拆分前的算子等价。
对于Conv(卷积)算子,则采用另外一种分配方案,即在Channel维度上进行拆分:即将大小B×H×W×C的输入Tensor拆分为 B×H×W×(C/N)的Tensor,同时将大小为H×W×C的参数Tensor拆分为N个H×W×(C/N)的参数Tensor。最终,我们将得到N个大小为B×H×W×(C/N)的输出Tensor,在Channel维度上将其重新合并为B×H×W×C的输出Tensor,这样保证了就保证了拆分后的算子与拆分前的算子等价。
执行器:每个工作节点上包含一个执行器进程,负责执行分配到这个节点上的(部分)深度学习模型的训练。执行器分为Worker和Parameter Server两种,分别完成模型的参数的训练和参数的汇总。
本实施例还提出了一种在HPC集群上自动实现对深度学习模型并行训练的方法,可以自动选择合适的算法对深度学习模型进行并行训练,并且最大可能的提升算法并行度,保证高效率完成深度学习的训练。图7是根据本实施例提供的交互流程图,包括:
步骤A.用户向应用管理器提交训练作业,指定待训练的深度学习模型,以及希望使用的节点数,应用管理器将其发送给资源管理器;
步骤B.资源管理器计算分配粒度G和数据并行度DP,并通过G和DP确定并行算法(数据并行、模型并行或者混合并行);根据G和DP在HPC上分配空闲节点给这个训练作业;
步骤C.应用管理器启动作业调度器,并传递用户提交的模型、资源管理器分配到资源及其参数;
步骤D.作业调度器根据分配粒度G,按照基于输入拆分或基于参数拆分方法,将模型拆分为G个子模型,并对G个子模型执行Split-Concat算子优化;
步骤E.作业调度器启动DP*G个执行器,每G个执行器组成一个执行组,以模型并行方式训练;数据被划分为DP个分片,在DP个执行组上以数据并行方式训练;
步骤F.所有的执行器执行完毕,应用管理器获取最终训练好的模型,并删除训练作业,释放资源。
采用本实施例的方案,能够在用户仅仅指定希望使用的GPU个数情况下,根据深度学习模型的特征、以及高性能集群的特征,自动化的产生与之对应的、高效的并行计算方案,实现既能节省算法研发投入,又能更快的训练模型的目的。
本发明的实施例还提供了一种存储介质,该存储介质中存储有计算机程序,其中,该计算机程序被设置为运行时执行上述任一项方法实施例中的步骤。
在一些实施例中,上述存储介质可以被设置为存储用于执行以下步骤的计算机程序:
S1,确定样本数据和可用的集群资源;
S2,将总训练模型拆分为多个子模型;
S3,使用集群资源在多个子模型上并行训练样本数据。
在一些实施例中,上述存储介质可以包括但不限于:U盘、只读存储器(Read-Only Memory,简称为ROM)、随机存取存储器(Random Access Memory,简称为RAM)、移动硬盘、磁碟或者光盘等各种可以存储计算机程序的介质。
本发明的实施例还提供了一种电子装置,包括存储器和处理器,该存储器中存储有计算机程序,该处理器被设置为运行计算机程序以执行上述任一项方法实施例中的步骤。
在一些实施例中,上述电子装置还可以包括传输设备以及输入输出设备,其中,该传输设备和上述处理器连接,该输入输出设备和上述处理器连接。
可以理解,存储器可以是易失性存储器或非易失性存储器,也可包括 易失性和非易失性存储器两者。其中,非易失性存储器可以是ROM、可编程只读存储器(PROM,Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM,Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM,Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM,ferromagnetic random access memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM,Compact Disc Read-Only Memory);磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM,Random Access Memory),其用作外部高速缓存。通过示例性但不是限制性说明,许多形式的RAM可用,例如静态随机存取存储器(SRAM,Static Random Access Memory)、同步静态随机存取存储器(SSRAM,Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM,Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM,Synchronous Dynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM,Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM,Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM,SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM,Direct Rambus Random Access Memory)。本发明实施例描述的存储器702旨在包括但不限于这些和任意其它适合类型的存储器。
上述本发明实施例揭示的方法可以应用于处理器中,或者由处理器实现。处理器可能是一种集成电路芯片,具有信号的处理能力。在实现过程中,上述方法的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器可以是通用处理器、数字信号处理器(DSP, Digital Signal Processor),或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。处理器可以实现或者执行本发明实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本发明实施例所公开的方法的步骤,可以直接体现为硬件译码处理器执行完成,或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中,该存储介质位于存储器,处理器读取存储器中的信息,结合其硬件完成前述方法的步骤。
在示例性实施例中,电子装置可以被一个或多个应用专用集成电路(ASIC,Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD,Programmable Logic Device)、复杂可编程逻辑器件(CPLD,Complex Programmable Logic Device)、现场可编程门阵列(FPGA,Field-Programmable Gate Array)、通用处理器、控制器、微控制器(MCU,Micro Controller Unit)、微处理器(Microprocessor)、或其他电子元件实现,用于执行前述方法。
在一些实施例中,上述处理器可以被设置为通过计算机程序执行以下步骤:
S1,确定样本数据和可用的集群资源;
S2,将总训练模型拆分为多个子模型;
S3,使用集群资源在多个子模型上并行训练样本数据。
本实施例中的具体示例可以参考上述实施例及可选实施方式中所描述的示例,本实施例在此不再赘述。
显然,本领域的技术人员应该明白,上述的本发明的各模块或各步骤可以用通用的计算装置来实现,它们可以集中在单个的计算装置上,或者分布在多个计算装置所组成的网络上,可选地,它们可以用计算装置可执行的程序代码来实现,从而,可以将它们存储在存储装置中由计算装置来执行,并且在某些情况下,可以以不同于此处的顺序执行所示出或描述的 步骤,或者将它们分别制作成各个集成电路模块,或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样,本发明不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已,并不用于限制本发明,对于本领域的技术人员来说,本发明可以有各种更改和变化。凡在本发明的原则之内,所作的任何修改、等同替换、改进等,均应包含在本发明的保护范围之内。
Claims (13)
- 一种数据训练方法,包括:确定样本数据和可用的集群资源;将总训练模型拆分为多个子模型;使用所述集群资源在所述多个子模型上并行训练所述样本数据。
- 根据权利要求1所述的方法,其中,所述将总训练模型拆分为多个子模型包括以下至少之一:将所述总训练模型拆分为多个第一子模型,其中,所述多个第一子模型并行连接;将所述总训练模型拆分为多个第二子模型,其中,所述多个第二子模型串行连接。
- 根据权利要求2所述的方法,其中,所述将所述总训练模型拆分为多个第一子模型,包括:根据算子的类型将所述总训练模型拆分为多个第一子模型,其中,所述总训练模型由一个或多个所述算子组成。
- 根据权利要求3所述的方法,其中,所述根据算子的类型将所述总训练模型拆分为多个第一子模型,包括:获取所述算子的类型,其中,所述算子的类型包括:稠密算子;将所述稠密算子拆分为N个输入为(B/N)×I的子稠密算子,其中,B为第一批维度,所述第一批维度的大小与指示的批尺寸的大小相同,I为所述稠密算子的输入向量的维度,N为大于1的整数,其中,所述第一子模型包括:所述子稠密算子。
- 根据权利要求3所述的方法,其中,所述根据算子的类型将所述总训练模型拆分为多个第一子模型,包括:获取所述算子的类型,其中,所述算子的类型包括:稠密算子;将所述稠密算子拆分为N个计算参数为I×(O/N)的子稠密算子,其中,所述子稠密算子与所述稠密算子的输入张量相同,O为所述稠密算子的输出向量的维度,I为所述稠密算子的输入向量的维度,N为大于1的整数,其中,所述第一子模型包括:所述子稠密算子。
- 根据权利要求3所述的方法,其中,所述根据算子的类型将所述总训练模型拆分为多个第一子模型,包括:获取所述算子的类型,其中,所述算子的类型包括:卷积算子;将所述卷积算子拆分为N个子卷积算子,其中,所述子卷积算子的输入张量为所述卷积算子的输入张量相同;其中,所述第一子模型包括:所述子卷积算子。
- 根据权利要求2所述的方法,其中,所述将所述总训练模型拆分为多个第二子模型,包括:解析所述总训练模型得到多个算子,其中,所述总训练模型包括Concat算子和Split算子,串行相邻的Concat算子和Split算子组成第一Concat-Split算子对;在所述第一Concat-Split算子对中,在所述Concat算子的输入张量与所述Split算子的输出张量相同时,在所述总训练模型中删除所述第一Concat-Split算子对中的Concat算子和Split算子后,将所述总训练模型拆分为多个第二子模型。
- 根据权利要求1所述的方法,其中,所述确定样本数据和可用的集群资源,包括:接收训练作业,从所述训练作业中获取对应的样本数据;确定集群当前空闲的第一处理器,以及接收指定的第二处理器信息,根据所述第二处理器信息在所述第一处理器中确定可用的处理器资源;其中,所述集群资源包括所述处理器资源。
- 根据权利要求1所述的方法,其中,所述使用所述集群资源在所述多个子模型上并行训练所述样本数据,包括:将所述样本数据划分为M个分片后,并行输入到所述集群资源的M*K个子模型上进行训练;其中,K为配置一个子模型所需要的最小集群资源,M为大于0的整数,K为大于0的整数。
- 一种数据训练装置,包括:确定模块,配置为确定样本数据和可用的集群资源;拆分模块,配置为将总训练模型拆分为多个子模型;训练模块,配置为使用所述集群资源在所述多个子模型上并行训练所述样本数据。
- 根据权利要求10所述的装置,其中,所述拆分模块包括以下至少之一:第一拆分单元,配置为将所述总训练模型拆分为多个第一子模型,其中,所述多个第一子模型并行连接;第二拆分单元,配置为将所述总训练模型拆分为多个第二子模型,其中,所述多个第二子模型串行连接。
- 一种存储介质,所述存储介质中存储有计算机程序,其中,所述计算机程序被设置为运行时执行所述权利要求1至9任一项中所述的方法。
- 一种电子装置,包括处理器和用于存储能够在处理器上运行的计算机程序的存储器,其中,所述处理器用于运行所述计算机程序时,执行权利要求1至9任一项所述方法的步骤。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/958,876 US20200342322A1 (en) | 2017-12-29 | 2018-11-06 | Method and device for training data, storage medium, and electronic device |
EP18897649.2A EP3734475A4 (en) | 2017-12-29 | 2018-11-06 | METHOD AND DEVICE FOR TRAINING DATA, STORAGE MEDIUM AND ELECTRONIC DEVICE |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711488171.3 | 2017-12-29 | ||
CN201711488171.3A CN109993299B (zh) | 2017-12-29 | 2017-12-29 | 数据训练方法及装置、存储介质、电子装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019128475A1 true WO2019128475A1 (zh) | 2019-07-04 |
Family
ID=67063035
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/114209 WO2019128475A1 (zh) | 2017-12-29 | 2018-11-06 | 数据训练方法及装置、存储介质、电子装置 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20200342322A1 (zh) |
EP (1) | EP3734475A4 (zh) |
CN (1) | CN109993299B (zh) |
WO (1) | WO2019128475A1 (zh) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110689121A (zh) * | 2019-09-24 | 2020-01-14 | 上海寒武纪信息科技有限公司 | 一种用多核处理器实现神经网络模型拆分方法及相关产品 |
CN111368941A (zh) * | 2020-04-10 | 2020-07-03 | 浙江大华技术股份有限公司 | 一种图像处理方法、装置以及计算机存储介质 |
CN112799834A (zh) * | 2021-01-26 | 2021-05-14 | 北京迈格威科技有限公司 | 训练数据分发方法、装置、电子设备及存储介质 |
CN113011585A (zh) * | 2021-03-19 | 2021-06-22 | 上海西井信息科技有限公司 | 消除拼接算子的编译优化方法、***、设备及存储介质 |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112396070A (zh) * | 2019-08-13 | 2021-02-23 | 中兴通讯股份有限公司 | 一种模型训练方法、装置和***,以及预测方法和装置 |
CN110503201A (zh) * | 2019-08-29 | 2019-11-26 | 苏州浪潮智能科技有限公司 | 一种神经网络分布式并行训练方法与装置 |
US20220383082A1 (en) * | 2019-09-24 | 2022-12-01 | Anhui Cambricon Information Technology Co., Ltd. | Neural network processing method and apparatus, computer device and storage medium |
CN110826708B (zh) * | 2019-09-24 | 2022-05-31 | 安徽寒武纪信息科技有限公司 | 一种用多核处理器实现神经网络模型拆分方法及相关产品 |
CN110689115B (zh) * | 2019-09-24 | 2023-03-31 | 安徽寒武纪信息科技有限公司 | 神经网络模型处理方法、装置、计算机设备及存储介质 |
US11604984B2 (en) * | 2019-11-18 | 2023-03-14 | Shanghai United Imaging Intelligence Co., Ltd. | Systems and methods for machine learning based modeling |
CN111309479B (zh) * | 2020-02-14 | 2023-06-06 | 北京百度网讯科技有限公司 | 一种任务并行处理的实现方法、装置、设备和介质 |
CN110929887B (zh) * | 2020-02-17 | 2020-07-03 | 支付宝(杭州)信息技术有限公司 | 逻辑回归模型训练方法、装置及*** |
CN111340237B (zh) * | 2020-03-05 | 2024-04-26 | 腾讯科技(深圳)有限公司 | 数据处理和模型运行方法、装置和计算机设备 |
CN112308205A (zh) * | 2020-06-28 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | 基于预训练模型的模型改进方法及装置 |
CN111782402A (zh) * | 2020-07-17 | 2020-10-16 | Oppo广东移动通信有限公司 | 数据处理方法、装置以及电子设备 |
KR20220064665A (ko) * | 2020-11-12 | 2022-05-19 | 삼성전자주식회사 | 인공지능 모델을 분산 처리하는 전자 장치 및 그 동작 방법 |
CN112508188B (zh) * | 2020-12-01 | 2024-06-14 | 北京奇艺世纪科技有限公司 | 一种分布式模型训练***、方法、装置、设备和存储介质 |
CN112884086B (zh) * | 2021-04-06 | 2022-08-30 | 北京百度网讯科技有限公司 | 模型训练方法、装置、设备、存储介质以及程序产品 |
CN114091685B (zh) * | 2021-11-08 | 2022-08-23 | 北京百度网讯科技有限公司 | 深度学习框架的张量切分方法、装置、设备和存储介质 |
CN114169427B (zh) * | 2021-12-06 | 2022-10-04 | 北京百度网讯科技有限公司 | 基于端到端自适应的分布式训练方法、装置、设备 |
CN114091029B (zh) * | 2022-01-24 | 2022-06-21 | 深信服科技股份有限公司 | 恶意文件检测模型的训练***、方法、设备、介质及平台 |
CN114943274B (zh) * | 2022-04-15 | 2024-07-19 | 支付宝(杭州)信息技术有限公司 | 模型训练方法、装置、存储介质、服务器、终端及*** |
CN116701001B (zh) * | 2023-08-08 | 2023-10-20 | 国网浙江省电力有限公司信息通信分公司 | 目标任务分配方法、装置、电子设备及存储介质 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106033554A (zh) * | 2015-03-13 | 2016-10-19 | 中国科学院声学研究所 | 一种基于分段的两级深度学习模型的大数据处理方法 |
CN106529682A (zh) * | 2016-10-28 | 2017-03-22 | 北京奇虎科技有限公司 | 一种在大数据集群中处理深度学习任务的方法和装置 |
US20170116520A1 (en) * | 2015-10-23 | 2017-04-27 | Nec Laboratories America, Inc. | Memory Efficient Scalable Deep Learning with Model Parallelization |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8201171B2 (en) * | 2007-06-27 | 2012-06-12 | Microsoft Corporation | Adjacent data parallel and streaming operator fusion |
KR20130119285A (ko) * | 2012-04-23 | 2013-10-31 | 한국전자통신연구원 | 클러스터 컴퓨팅 환경에서의 자원 할당 장치 및 그 방법 |
US9508347B2 (en) * | 2013-07-10 | 2016-11-29 | Tencent Technology (Shenzhen) Company Limited | Method and device for parallel processing in model training |
US9639642B2 (en) * | 2013-10-09 | 2017-05-02 | Fujitsu Limited | Time series forecasting ensemble |
US9672474B2 (en) * | 2014-06-30 | 2017-06-06 | Amazon Technologies, Inc. | Concurrent binning of machine learning data |
CN104933463B (zh) * | 2015-07-07 | 2018-01-23 | 杭州朗和科技有限公司 | 深度神经网络模型的训练方法和设备 |
US10853130B1 (en) * | 2015-12-02 | 2020-12-01 | Color Genomics, Inc. | Load balancing and conflict processing in workflow with task dependencies |
US10354201B1 (en) * | 2016-01-07 | 2019-07-16 | Amazon Technologies, Inc. | Scalable clustering for mixed machine learning data |
CN107025205B (zh) * | 2016-01-30 | 2021-06-22 | 华为技术有限公司 | 一种分布式***中的训练模型的方法及设备 |
WO2018126073A1 (en) * | 2016-12-30 | 2018-07-05 | Lau Horace H | Deep learning hardware |
CN107316082A (zh) * | 2017-06-15 | 2017-11-03 | 第四范式(北京)技术有限公司 | 用于确定机器学习样本的特征重要性的方法及*** |
CN107480717A (zh) * | 2017-08-16 | 2017-12-15 | 北京奇虎科技有限公司 | 训练作业处理方法及***、计算设备、计算机存储介质 |
-
2017
- 2017-12-29 CN CN201711488171.3A patent/CN109993299B/zh active Active
-
2018
- 2018-11-06 EP EP18897649.2A patent/EP3734475A4/en active Pending
- 2018-11-06 US US16/958,876 patent/US20200342322A1/en active Pending
- 2018-11-06 WO PCT/CN2018/114209 patent/WO2019128475A1/zh unknown
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106033554A (zh) * | 2015-03-13 | 2016-10-19 | 中国科学院声学研究所 | 一种基于分段的两级深度学习模型的大数据处理方法 |
US20170116520A1 (en) * | 2015-10-23 | 2017-04-27 | Nec Laboratories America, Inc. | Memory Efficient Scalable Deep Learning with Model Parallelization |
CN106529682A (zh) * | 2016-10-28 | 2017-03-22 | 北京奇虎科技有限公司 | 一种在大数据集群中处理深度学习任务的方法和装置 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3734475A4 |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110689121A (zh) * | 2019-09-24 | 2020-01-14 | 上海寒武纪信息科技有限公司 | 一种用多核处理器实现神经网络模型拆分方法及相关产品 |
EP3979143A4 (en) * | 2019-09-24 | 2023-02-08 | Anhui Cambricon Information Technology Co., Ltd. | METHOD OF PERFORMING A SPLIT IN A NEURAL NETWORK MODEL USING A MULTI-CORE PROCESSOR AND RELATIVE PRODUCT |
CN111368941A (zh) * | 2020-04-10 | 2020-07-03 | 浙江大华技术股份有限公司 | 一种图像处理方法、装置以及计算机存储介质 |
CN111368941B (zh) * | 2020-04-10 | 2023-09-01 | 浙江大华技术股份有限公司 | 一种图像处理方法、装置以及计算机存储介质 |
CN112799834A (zh) * | 2021-01-26 | 2021-05-14 | 北京迈格威科技有限公司 | 训练数据分发方法、装置、电子设备及存储介质 |
CN112799834B (zh) * | 2021-01-26 | 2024-05-07 | 北京迈格威科技有限公司 | 训练数据分发方法、装置、电子设备及存储介质 |
CN113011585A (zh) * | 2021-03-19 | 2021-06-22 | 上海西井信息科技有限公司 | 消除拼接算子的编译优化方法、***、设备及存储介质 |
CN113011585B (zh) * | 2021-03-19 | 2023-09-26 | 上海西井科技股份有限公司 | 消除拼接算子的编译优化方法、***、设备及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
US20200342322A1 (en) | 2020-10-29 |
EP3734475A4 (en) | 2021-10-20 |
CN109993299B (zh) | 2024-02-27 |
EP3734475A1 (en) | 2020-11-04 |
CN109993299A (zh) | 2019-07-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2019128475A1 (zh) | 数据训练方法及装置、存储介质、电子装置 | |
US12032573B2 (en) | Structured cluster execution for data streams | |
JP6726246B2 (ja) | 畳み込みニューラルネットワークにおいて演算を実行する方法および装置並びに非一時的な記憶媒体 | |
EP3267310A1 (en) | Data processing method and device | |
DE102016118210A1 (de) | Granulare Dienstqualität für Computer-Ressourcen | |
CN105956666B (zh) | 一种机器学习方法及*** | |
CN108205469B (zh) | 一种基于MapReduce的资源分配方法及服务器 | |
US11107187B2 (en) | Graph upscaling method for preserving graph properties | |
Elsayed et al. | Mapreduce: State-of-the-art and research directions | |
TWI603203B (zh) | 分散式計算之運算參數與叢集系統組態的推薦方法與系統 | |
TWI775210B (zh) | 用於卷積運算的資料劃分方法及處理器 | |
CN105975493A (zh) | 一种文件合并方法及装置 | |
Berlińska et al. | Comparing load-balancing algorithms for MapReduce under Zipfian data skews | |
US11194623B2 (en) | Resource scheduling method and related apparatus | |
CN113222125A (zh) | 卷积运算方法及芯片 | |
TWI758223B (zh) | 具有動態最小批次尺寸之運算方法,以及用於執行該方法之運算系統及電腦可讀儲存媒體 | |
KR20140103805A (ko) | 데이터 분배 방법 및 장치 | |
US10592473B2 (en) | Method for improving energy efficiency of map-reduce system and apparatus thereof | |
CN113222099A (zh) | 卷积运算方法及芯片 | |
CN114328606B (zh) | Sql执行效率的提高方法、设备及存储介质 | |
CN105740073A (zh) | 一种动态控制操作***进程数量的方法及装置 | |
US11960868B2 (en) | Branch objects for dependent optimization problems | |
WO2015143708A1 (zh) | 后缀数组的构造方法及装置 | |
CN110415162B (zh) | 大数据中面向异构融合处理器的自适应图划分方法 | |
CN113608724A (zh) | 一种基于模型缓存实现的离线仓库实时交互方法与*** |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18897649 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2018897649 Country of ref document: EP Effective date: 20200729 |