WO2023045638A1 - 计算装置、利用计算装置实施卷积运算的方法及相关产品 - Google Patents
计算装置、利用计算装置实施卷积运算的方法及相关产品 Download PDFInfo
- Publication number
- WO2023045638A1 WO2023045638A1 PCT/CN2022/113302 CN2022113302W WO2023045638A1 WO 2023045638 A1 WO2023045638 A1 WO 2023045638A1 CN 2022113302 W CN2022113302 W CN 2022113302W WO 2023045638 A1 WO2023045638 A1 WO 2023045638A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- circuit
- data
- convolution
- storage
- output
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 194
- 238000012545 processing Methods 0.000 claims abstract description 584
- 238000003860 storage Methods 0.000 claims abstract description 433
- 239000000872 buffer Substances 0.000 claims description 300
- 238000004364 calculation method Methods 0.000 claims description 178
- 230000008569 process Effects 0.000 claims description 95
- 230000036961 partial effect Effects 0.000 claims description 62
- 238000009826 distribution Methods 0.000 claims description 33
- 238000009825 accumulation Methods 0.000 claims description 30
- 230000000903 blocking effect Effects 0.000 claims description 13
- 230000003139 buffering effect Effects 0.000 claims description 12
- 101150083561 nop-1 gene Proteins 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 62
- 210000002569 neuron Anatomy 0.000 description 45
- 210000002364 input neuron Anatomy 0.000 description 37
- 230000015654 memory Effects 0.000 description 23
- 230000002441 reversible effect Effects 0.000 description 20
- 238000003062 neural network model Methods 0.000 description 15
- 238000013135 deep learning Methods 0.000 description 12
- 230000005540 biological transmission Effects 0.000 description 11
- 239000011159 matrix material Substances 0.000 description 8
- 238000012549 training Methods 0.000 description 8
- 210000004205 output neuron Anatomy 0.000 description 7
- 230000001131 transforming effect Effects 0.000 description 7
- 238000013500 data storage Methods 0.000 description 6
- 238000005457 optimization Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 102100030148 Integrator complex subunit 8 Human genes 0.000 description 4
- 101710092891 Integrator complex subunit 8 Proteins 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 102100025297 Mannose-P-dolichol utilization defect 1 protein Human genes 0.000 description 3
- 101710089919 Mannose-P-dolichol utilization defect 1 protein Proteins 0.000 description 3
- 239000003086 colorant Substances 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 235000019800 disodium phosphate Nutrition 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000005481 NMR spectroscopy Methods 0.000 description 1
- 240000007594 Oryza sativa Species 0.000 description 1
- 235000007164 Oryza sativa Nutrition 0.000 description 1
- 101100070120 Xenopus laevis has-rs gene Proteins 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 235000009566 rice Nutrition 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/15—Correlation function computation including computation of convolution operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
- G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
- G06F7/544—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices for evaluating functions by calculation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Definitions
- the present disclosure relates generally to the field of data processing. More specifically, the present disclosure relates to a computing device configured to perform convolution operations, a method for implementing convolution operations using the computing device, a chip, and a board.
- Deep learning Deep Learning
- AI artificial intelligence
- Neural network is one of the most critical technologies in artificial intelligence and deep learning, among which Convolution Neural Network (CNN) is the most important network type.
- the most critical calculation in the convolutional neural network is the convolution operation (Convolution Operation) of the convolution layer (Conv layer).
- the function of the convolutional layer is to extract features from the input data. Through multi-layer convolution, complex features can be extracted to ensure that the network has sufficient expressive ability and generalization ability.
- the neural network model contains a large number of various types of convolution operations, and the computing performance of the convolution operation greatly affects the computing performance of the entire neural network model.
- the corresponding input feature maps and weights may have different dimensions.
- the present disclosure proposes a computing device in various aspects, which can make the The data can be adapted to the hardware of the convolution operation, thereby improving the computational efficiency of the convolution operation.
- the convolution operation in the embodiment of the present disclosure can be an operation in various neural network models, and these neural network models can be applied in various fields, such as image processing, speech processing, text processing, etc., such processing can include but not limited to identification and classification.
- an embodiment of the present disclosure provides a computing device configured to perform a convolution operation, the computing device including a main processing circuit configured to: obtain an input feature map and/or convolution The product kernel, wherein the input feature map and the convolution kernel have been split into multiple split units according to the convolution split scheme and their dimension storage order is converted, wherein the convolution split scheme is based on the input feature
- the size of the lowest storage dimension before graph splitting is determined, the convolution splitting scheme indicates the shape of the splitting unit, the amount of data contained in a splitting unit does not exceed the maximum single operation of the hardware, and a splitting The data in the unit is continuously stored as a data row; and a plurality of slave processing circuits are used to perform convolution operations on the input feature map and corresponding split units of the convolution kernel.
- an embodiment of the present disclosure provides a chip, which includes the computing device in any embodiment of the foregoing first aspect.
- an embodiment of the present disclosure provides a board, which includes the chip in any embodiment of the foregoing second aspect.
- an embodiment of the present disclosure provides a method for performing a convolution operation by the computing device in any embodiment of the first aspect.
- the solution of the embodiment of the present disclosure applies different convolution splitting schemes to input feature maps of different dimensions to adapt to hardware operations
- the processing capability of the device can fully utilize the parallel processing capability of multiple slave processing circuits, which can effectively improve the operation efficiency of the convolution operation.
- input feature maps and weights can be transmitted through different data paths, thereby supporting multiple multiplexing of input feature maps and weights, further optimizing convolution operations, and reducing data memory access.
- Fig. 1 shows the structural diagram of the board card of the disclosed embodiment
- FIG. 2 shows a structural diagram of a combination processing device according to an embodiment of the present disclosure
- FIG. 3 shows a schematic diagram of the internal structure of a processor core of a single-core or multi-core computing device according to an embodiment of the disclosure
- Figures 4a-4c show examples of several exemplary convolution operation principles that can be applied to embodiments of the present disclosure
- Fig. 5 shows a schematic structural block diagram of a computing device according to an embodiment of the disclosure
- FIG. 6 shows an exemplary data storage sequence according to an embodiment of the present disclosure
- Figures 7a-7d illustrate several exemplary grouping modes according to embodiments of the present disclosure
- Fig. 8 shows an exemplary split schematic diagram of an input feature map according to an embodiment of the present disclosure
- FIGS. 9a-9d show schematic diagrams of data storage in a second storage circuit according to an embodiment of the present disclosure.
- 10a-10b show a schematic diagram of division of output points of an arithmetic circuit according to an embodiment of the present disclosure
- FIG. 11 shows a schematic diagram of splitting and storage of the Forward16 scheme according to an embodiment of the present disclosure
- FIG. 12 shows a schematic diagram of a single operation in the Forward16 scheme according to an embodiment of the present disclosure
- Fig. 13 shows a schematic diagram of sliding convolution in the Forward16 scheme according to an embodiment of the present disclosure
- FIG. 14 shows a schematic diagram of accumulation of sliding convolution results in the Forward16 scheme according to an embodiment of the present disclosure
- Fig. 15 shows a schematic diagram of the output data format of the Forward16 splitting scheme according to an embodiment of the present disclosure
- FIG. 16 shows a schematic diagram of splitting and storage of the Forward4 scheme according to an embodiment of the present disclosure
- FIG. 17 shows a schematic diagram of a single operation in the Forward4 scheme according to an embodiment of the disclosure.
- Fig. 18 shows a schematic diagram of sliding convolution in the Forward4 scheme according to an embodiment of the present disclosure
- Fig. 19 shows a schematic diagram of the output data format of the Forward4 scheme according to an embodiment of the present disclosure
- FIG. 20 shows a schematic diagram of division of output points of the computing circuit in the Forward1 scheme according to an embodiment of the present disclosure
- FIG. 21 shows a schematic diagram of a single operation in the Forward1 scheme according to an embodiment of the present disclosure
- Fig. 22 shows a schematic diagram of sliding convolution in the Forward1 scheme according to an embodiment of the present disclosure
- Fig. 23 shows a schematic diagram of the output data format of the Forward1 scheme according to an embodiment of the present disclosure
- FIG. 24 shows a schematic diagram of data storage in the second storage circuit in the Update1 scheme according to an embodiment of the disclosure
- Fig. 25 shows a schematic diagram of sliding convolution in the Update1 scheme according to an embodiment of the disclosure
- Fig. 26 shows a schematic diagram of the output data format of the Update1 scheme according to an embodiment of the present disclosure
- 27a-27b show exemplary storage contents in the second storage circuit in different grouping modes in the Update4 scheme according to an embodiment of the present disclosure
- FIG. 28 shows a schematic diagram of a single operation process in the Update4 solution according to an embodiment of the present disclosure
- Figure 29 shows a schematic diagram of the sliding convolution process in the Update4 scheme according to an embodiment of the disclosure.
- FIG. 30 shows a schematic diagram of an output data format of the Update4 scheme according to an embodiment of the present disclosure.
- the term “if” may be interpreted as “when” or “once” or “in response to determining” or “in response to detecting” depending on the context.
- FIG. 1 shows a schematic structural diagram of a board 10 according to an embodiment of the present disclosure.
- the board card 10 includes a chip 101, which is a system-on-chip (System on Chip, SoC), or system-on-chip, integrated with one or more combined processing devices, and the combined processing device is an artificial
- SoC System on Chip
- the intelligent computing unit is used to support various deep learning and machine learning algorithms to meet the intelligent processing requirements in complex scenarios in the fields of computer vision, speech, natural language processing, and data mining.
- deep learning technology is widely used in the field of cloud intelligence.
- a notable feature of cloud intelligence applications is the large amount of input data, which has high requirements for the storage capacity and computing power of the platform.
- the board 10 of this embodiment is suitable for cloud intelligence applications. applications, with huge off-chip storage, on-chip storage and powerful computing capabilities.
- the chip 101 is connected to an external device 103 through an external interface device 102 .
- the external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card or a wifi interface, and the like.
- the data to be processed can be transmitted to the chip 101 by the external device 103 through the external interface device 102 .
- the calculation result of the chip 101 can be sent back to the external device 103 via the external interface device 102 .
- the external interface device 102 may have different interface forms, such as a PCIe interface and the like.
- the board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105 .
- the storage device 104 is connected and data transmitted with the control device 106 and the chip 101 through the bus.
- the control device 106 in the board 10 is configured to regulate the state of the chip 101 .
- the control device 106 may include a microcontroller (Micro Controller Unit, MCU).
- FIG. 2 is a block diagram showing the combined processing means in the chip 101 of this embodiment.
- the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .
- the computing device 201 is configured to perform operations specified by the user, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor for performing deep learning or machine learning calculations, which can interact with the processing device 203 through the interface device 202 to Work together to complete user-specified operations.
- the interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203 .
- the computing device 201 may obtain input data from the processing device 203 via the interface device 202 and write it into a storage device on the computing device 201 .
- the computing device 201 may obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the chip of the computing device 201 .
- the interface device 202 may also read data in the storage device of the computing device 201 and transmit it to the processing device 203 .
- the processing device 203 performs basic control including but not limited to data transfer, starting and/or stopping the computing device 201 .
- the processing device 203 may be one or more types of a central processing unit (central processing unit, CPU), a graphics processing unit (graphics processing unit, GPU) or other general-purpose and/or special-purpose processors.
- processors including but not limited to digital signal processors (digital signal processors, DSPs), application specific integrated circuits (application specific integrated circuits, ASICs), field-programmable gate arrays (field-programmable gate arrays, FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and the number thereof can be determined according to actual needs.
- the computing device 201 of the present disclosure can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when considering the integration of the computing device 201 and the processing device 203 together, they are considered to form a heterogeneous multi-core structure.
- the storage device 204 is used to store data to be processed, which may be a DRAM, which is a DDR memory, and its size is usually 16G or larger, and is used to store data of the computing device 201 and/or the processing device 203 .
- FIG. 3 shows a schematic diagram of the internal structure of a processing core when the computing device 201 is a single-core or multi-core device.
- the computing device 301 is used for processing input data such as computer vision, speech, natural language, data mining, etc.
- the computing device 301 includes three modules: a control module 31 , an operation module 32 and a storage module 33 .
- the control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the task of deep learning, which includes an instruction fetch unit (IFU) 311 and an instruction decoding unit (instruction decode unit, IDU) 312.
- the instruction fetching unit 311 is used to obtain instructions from the processing device 203 , and the instruction decoding unit 312 decodes the obtained instructions and sends the decoding results to the computing module 32 and the storage module 33 as control information.
- the operation module 32 includes a vector operation unit 321 and a matrix operation unit 322 .
- the vector operation unit 321 is used to perform vector operations, and can support complex operations such as vector multiplication, addition, and nonlinear transformation;
- the matrix operation unit 322 is responsible for the core calculation of the deep learning algorithm, namely matrix multiplication and convolution.
- the storage module 33 is used to store or transfer relevant data, including a neuron storage unit (neuron RAM, NRAM) 331, a weight storage unit (weight RAM, WRAM) 332, and a direct memory access module (direct memory access, DMA) 333.
- NRAM 331 is used to store input neurons, output neurons and intermediate results after calculation;
- WRAM 332 is used to store convolution kernels of deep learning networks, that is, weights;
- DMA 333 is connected to DRAM 204 through bus 34, responsible for computing device 301 Data transfer between DRAM 204 and DRAM 204.
- an embodiment of the present disclosure provides a computing device configured to perform a convolution operation, so that the convolution operation in a neural network model, for example, can be optimized.
- the convolutional layer in a neural network model can perform convolution operations by applying convolution kernels (also called filters, weights, etc.) to input feature maps (also called input data, neurons, or input neurons) processing for feature extraction.
- the convolution layer can contain multiple convolution kernels, and each element that makes up the convolution kernel corresponds to a weight coefficient and a bias.
- the neural network model may contain various convolution operation layers, such as convolution layers that perform forward and conventional 3D convolution operations, and deconvolution layers that perform depthwise convolution operations. In reverse training, it may be necessary to perform reverse depthwise convolution operations or cross-product convolution operations. Embodiments of the present disclosure can be optimized for these different types of convolution operations.
- X is the input data
- Y is the output data
- K is the convolution kernel
- Kh and Kw are the length and width of K
- sh and sw are the strides in the length and width directions
- the formula ignores Bias, fill pad and dilation, and assume that the input data X has been filled, and the convolution kernel has been expanded.
- the formula ignores the N dimension and the C dimension.
- the forward calculation of the neural network model is independent in the N dimension and fully connected in the C dimension.
- the results of the multiplication of the H, W, and Ci directions are accumulated, so it is called 3D convolution.
- the Ci dimension of the convolution kernel is equal to the Ci dimension of the input feature map, so the convolution kernel does not slide in the Ci direction, which is a pseudo 3D convolution.
- the above convolution operations are called 3D convolution operations.
- Fig. 4a shows an example of an exemplary conventional 3D convolution operation principle to which embodiments of the present disclosure can be applied.
- the figure exemplarily shows four-dimensional input data X with a size of [N Hi Wi Ci], which can be expressed as N three-dimensional rectangles 410a of size Hi ⁇ Wi ⁇ Ci.
- the figure also exemplarily shows a four-dimensional convolution kernel K with a size of [Co Kh Kw Ci], which can be expressed as Co three-dimensional convolution kernels 420a of size Kh ⁇ Kw ⁇ Ci.
- the convolution result of the input data X and the convolution kernel K obtains the output data Y, which is four-dimensional data of the size [N Ho Wo Co], which can be expressed as N three-dimensional rectangles 430a of the size Ho ⁇ Wo ⁇ Co.
- the figure also specifically shows an example of a convolution operation, in which the input data is an input feature map 440a with a size of 6 ⁇ 6 ⁇ 3, and the N dimension is omitted; the convolution kernel is a three-dimensional convolution kernel 450a with a size of 3 ⁇ 3 ⁇ 3 , for a single convolution kernel Co; the output data is a 4 ⁇ 4 output feature map 460a.
- the specific operation process is as follows:
- the convolution kernel 450a scans the input feature map 440a according to a certain step size, performs matrix element multiplication and summation on the input features in the convolution window 470a, and superimposes the deviation. That is, the value at each position in the output feature map 460a is obtained by performing a two-dimensional convolution operation on the corresponding block and the corresponding convolution kernel of each input feature map and then summing them up. For example, the figure shows that the value at (0,0) on the output feature map 460a (that is, the convolution output point) is double-converted by the convolution window 470a framed by the black cube in the input feature map and the three-dimensional convolution kernel 450a. The three-dimensional convolution operation obtains 3 values, which are summed to obtain the final value.
- the position of the convolution kernel 450a can be moved on the input feature map 440a, that is, the convolution window of the convolution output point can be moved.
- the convolution step size (Sx, Sy) is (1,1).
- the convolution operation can be obtained respectively The value at (0,1) or (1,0) position on the feature map 460a is output.
- a convolutional layer of the neural network there are N groups of input feature maps, and each group contains Hi ⁇ Wi ⁇ Ci information, where Hi and Wi are the height and width of the input feature map, and Ci is The number of input feature maps, also known as the number of input channels.
- the convolutional layer has Ci ⁇ Co convolution kernels of Kh ⁇ Kw size, where Ci is the number of input channels, Co is the number of output feature maps (or the number of output channels), Kh and Kw are the height and width.
- the output feature map contains Ho ⁇ Wo ⁇ Co pieces of information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels.
- the convolution step size Sx, Sy
- the size of the convolution step size will affect the size of the output feature map.
- Fig. 4b shows an example of an exemplary depthwise convolution operation principle to which embodiments of the present disclosure can be applied.
- each convolution kernel needs to be calculated and accumulated with all layers (input channels) of the input feature map, so the number of input channels of each convolution kernel is equal to the number of input channels of the input feature map.
- each convolution kernel is single-channel, one convolution kernel is responsible for one channel, and one channel is only convolved by one convolution kernel. Therefore, depth convolution is sometimes called 2D convolution, that is, only sliding and accumulating in H and W dimensions.
- the dimension of the input feature map 410b is 12 ⁇ 12 ⁇ 3, that is, it includes three channels, and each channel includes a 12 ⁇ 12 image.
- Three convolution kernels 420b are respectively used in this depthwise convolution, and each convolution kernel is single-channel, and its size is, for example, 5 ⁇ 5 ⁇ 1.
- Each convolution kernel only convolutes one channel of the input feature map 410b, and such convolution produces an output of size 8 ⁇ 8 ⁇ 1 each time, and then these outputs are stacked together to create an 8 ⁇ 8 ⁇ 3 image, and finally obtain an output feature map 430b with a size of 8 ⁇ 8 ⁇ 3. It can be seen from the figure that the depth (number of channels) of the output feature map remains consistent with the input feature map.
- the dimensions of its input feature map, convolution kernel and output feature map can be simplified to C (channel), H (height), W ( width) in three dimensions.
- top_diff and bottom_diff are neuron gradients
- W is the weight of this iteration
- ⁇ W is the weight gradient of this iteration
- top_diff is similar to the operation process between the input neuron and the weight W, where top_diff is equivalent to the input feature map.
- top_diff is equivalent to the convolution kernel, and slides and accumulates in the XY direction of bottom_data.
- the operation principle can refer to Figure 4b.
- the sizes of top_diff and bottom_data are usually relatively large. Therefore, the embodiments of the present disclosure also provide an optimization solution for the convolution operation (reverse depthwise convolution for short) in this scenario.
- the operations in the reverse process can be called cross-product convolution operations.
- the embodiments of the present disclosure can also provide an optimization solution for this convolution operation.
- Fig. 4c shows an example of the principle of cross-product convolution operation that can be applied to the embodiments of the present disclosure.
- the figure shows an example of the three-dimensional data top_diff whose size is [Ho Wo Co], which can be expressed as a three-dimensional rectangle 410c of the size Ho ⁇ Wo ⁇ Co; the figure also shows the three-dimensional data whose size is [Hi Wi Ci] bottom_data, which can be expressed as a three-dimensional rectangle 420c with the size of Hi ⁇ Wi ⁇ Ci.
- the top_diff and bottom_data perform the cross-product convolution operation to obtain the output data 430c, which is four-dimensional data with a size of [Co Kh Kw Ci], which can be expressed as Co three-dimensional rectangles 430c with the size of Kh ⁇ Kw ⁇ Ci.
- Ci copies are copied to obtain the data 440c of Ho ⁇ Wo ⁇ Ci.
- Perform depth convolution operation on the data 440c and bottom_data (refer to the schematic diagram of FIG. 4b ), that is, do not accumulate in the Ci direction, so as to obtain an output 460c, which is three-dimensional data of Kh ⁇ Kw ⁇ Ci size.
- For each HoWo surface repeat the copying and depth convolution operation to obtain Co three-dimensional data of Kh ⁇ Kw ⁇ Ci size, that is, obtain a four-dimensional convolution kernel 430c, whose size is Co ⁇ Kh ⁇ Kw ⁇ Ci .
- input feature map (Feature map), input data, neuron or input neuron are used interchangeably; convolution kernel, filter or weight are used interchangeably.
- H (height) and Y dimensions are used interchangeably, and the W (width) and X dimensions are used interchangeably.
- the H dimension of the input feature map can be expressed as Hi or Yi
- the H dimension of the output feature map can be expressed as Ho or Yo
- the W dimension can be expressed similarly.
- each convolution output point has a corresponding convolution window, and the shape of the convolution window is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the result of the multiplication and accumulation of the input feature map and the weight in its convolution window.
- the data involved can be divided into input feature maps, convolution kernels, and output feature maps.
- top_diff corresponds to the convolution kernel
- bottom_data corresponds to the input feature map
- ⁇ W corresponds to the output feature map.
- a computing device with a master-slave structure may be used to implement the above convolution operation.
- different data paths can be configured for input feature maps and convolution kernels, thereby improving memory access efficiency.
- FIG. 5 shows a schematic structural block diagram of a computing device 500 according to an embodiment of the disclosure. It can be understood that this structure can be regarded as the refinement of the internal structure of the operation module of a single processing core in FIG. 3 , or can be regarded as a functional division block diagram based on the combination of multiple operation modules of the processing core shown in FIG. 3 .
- a computing device 500 in an embodiment of the present disclosure may be configured to perform various types of convolution operations, which may include a master processing circuit (MA) 510 and a plurality of slave processing circuits (SL) 520, shown in the figure 16 slave processing circuits SL0 to SL15 are shown.
- MA master processing circuit
- SL slave processing circuits
- the master processing circuit and the slave processing circuits, as well as multiple slave processing circuits, can communicate with each other through various connections.
- the connection between multiple slave processing circuits can be hard-wired, or logically configured according to, for example, micro-instructions to form a variety of slave processing circuits
- the topology of the array Embodiments of the present disclosure are not limited in this regard.
- the main processing circuit and the slave processing circuit can cooperate with each other, thereby realizing parallel operation processing.
- the main processing circuit and the slave processing circuit may include various calculation circuits, for example, may include a vector operation unit and a matrix operation unit.
- the vector operation unit is used to perform vector operations, and can support complex operations such as vector multiplication, addition, and nonlinear transformation;
- the matrix operation unit is responsible for the core calculations of deep learning algorithms, such as matrix multiplication and convolution.
- the slave processing circuit can be used to perform intermediate operations on corresponding data in parallel according to the operation instruction to obtain multiple intermediate results, and transmit the multiple intermediate results back to the main processing circuit.
- the computing device 500 By setting the computing device 500 into a master-slave structure (for example, a master-multiple-slave structure, or a multi-master-multiple-slave structure, the present disclosure is not limited in this respect), for the calculation instructions of the forward operation, the data can be disassembled according to the calculation instructions. In this way, multiple slave processing circuits are used to perform parallel calculations on the part with a large amount of calculation to improve the calculation speed, save calculation time, and reduce power consumption.
- a master-slave structure for example, a master-multiple-slave structure, or a multi-master-multiple-slave structure, the present disclosure is not limited in this respect
- multiple multiplexing methods of input feature maps and weights can be supported, thereby reducing the amount of data access during operations and improving processing efficiency .
- the computing device 500 may further include a first storage circuit 530 and a second storage circuit 540 for respectively storing data transmitted via different data channels.
- the first storage circuit 530 and the second storage circuit 540 may be two storage blocks formed by dividing the same memory, or may be two independent memories, which are not specifically limited here.
- the first storage circuit 530 can be used to store multicast data, that is, the data in the first storage circuit will be transmitted to multiple slave processing circuits through the broadcast bus, and these slave processing circuits receive the same data. It can be understood that broadcasting and multicasting can be implemented through the broadcasting bus. Multicast refers to a communication method that transmits a piece of data to multiple slave processing circuits; broadcasting is a communication method that transmits a piece of data to all slave processing circuits, which is a special case of multicast. Since both multicast and broadcast correspond to a one-to-many transmission mode, there is no special distinction between the two in this document. Broadcast and multicast can be collectively referred to as multicast, and those skilled in the art can clarify their meanings according to the context.
- the second storage circuit 540 may be used to store and distribute data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.
- one of the input feature map and the convolution kernel can be determined as multicast data and stored in the first storage circuit, so as to transmit the data to a plurality of scheduled slave processing circuits by broadcasting during operation .
- the other of the input feature map and the convolution kernel may be determined as distribution data and stored in the second storage circuit. These distributed data can be distributed to corresponding slave processing circuits before operation.
- FIG. 5 also shows a schematic diagram of the internal structure of the slave processing circuit SL according to an embodiment of the present disclosure.
- each slave processing circuit 520 may include a plurality of operation circuits CU 521, a first buffer circuit 522 and a second buffer circuit 523.
- four arithmetic circuits CU0 to CU3 are shown.
- the number of computing circuits may be more or less depending on specific hardware configurations, and the embodiments of the present disclosure are not limited in this respect.
- the first buffer circuit 522 may be used for buffering weights or input feature maps assigned to the slave processing circuit.
- the second buffer circuit 523 may be used for buffering the input feature map or the weight assigned to the slave processing circuit. These two buffer circuits are used to select the data involved in the operation.
- the data of the first buffer circuit 522 can be a plurality of data rows from, for example, the first storage circuit 530 or the second storage circuit 540, and correspondingly, the data of the second buffer circuit 523 can come from, for example, the second storage circuit 540 or the first storage circuit 540 Multiple data rows of circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding computing circuit CU 521 or broadcast to all CUs 521 in the slave processing circuit 520 during the operation.
- Each operation circuit CU521 is used to perform bitwise multiply-accumulate operations on the data rows selected from the first buffer circuit and the data rows selected from the second buffer circuit in each operation cycle.
- the slave processing circuit 520 may also include a third buffer circuit 524 for buffering the calculation results of each calculation circuit CU 521.
- each processing circuit and storage circuit are shown as separate modules in FIG. 5 , according to different configurations, the storage circuit and the processing circuit may also be combined into one module.
- the first storage circuit 530 can be combined with the main processing circuit 510
- the second storage circuit 540 can be shared by multiple slave processing circuits 520, and an independent storage area is assigned to each slave processing circuit to speed up access.
- Embodiments of the present disclosure are not limited in this regard.
- the main processing circuit and the slave processing circuit may belong to different modules of the same processor or chip, or may belong to different processors, and the present disclosure is not limited in this respect.
- the dimensions of the involved multidimensional data are represented by (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in the memory. It can be understood that although the multidimensional data has multiple dimensions, since the layout of the memory is always one-dimensional, there is a corresponding relationship between the multidimensional data and the storage order on the memory. Multidimensional data is usually allocated in continuous storage space, that is, multidimensional data can be expanded in one dimension and stored in the memory in sequence.
- the initial input feature map can be stored sequentially in a low-dimensional (where C/Ci is the lowest dimension) priority mode; and in order to optimize the convolution operation, during the operation or before the operation
- the storage order of the input feature maps can be adjusted, as will be described in detail later.
- Adjacent dimensions refer to dimensions that are next to each other in the dimension information representation of multidimensional data, for example, W and Ci are adjacent, and adjacent dimensions may also be called continuous dimensions.
- the main computing unit of the hardware is a vector multiply-accumulate operator.
- Implementing support for various convolution algorithms in hardware design is essentially to extract the multiplication and addition operations in the algorithm to the maximum extent, and realize the connection between the on-chip RAM (such as NRAM, WRAM, etc. in Figure 3) and the arithmetic unit through the data path. efficiently exchange the input and output data of the multiply-accumulate operation.
- Hardware is stored line by line (cache line).
- the read, write, and calculation operations are most efficient when the entire line is aligned. Therefore, in order to make full use of the bandwidth and adapt to the memory access requirements of the arithmetic unit array, it is usually necessary to
- the data is vectorized and aligned.
- the design of artificial intelligence chips usually takes the Ci dimension as the lowest dimension, that is, the above-mentioned NHWC arrangement order, and the data on the Ci dimension is continuous. Therefore, vectorization alignment requires that the size of the Ci dimension be aligned to a specified value, such as the alignment value M, so that the number of accesses is performed in units of the alignment value M, and M can also be called the maximum single operation of the hardware.
- M can have different values, such as 64bit, 128bit, 256bit, 512bit, etc.
- the size of the input port of the operator array is also related to M.
- the input port size of the operator array is usually twice the size of M, that is, the input of the alignment value M scale is processed at one time.
- Feature map data and weight data When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.
- the Ci dimension of the input feature map is small, such as smaller than the size of a cache line, the Ci dimension needs to be filled to one line of data (for example, 512 bits), that is, invalid data 0 is filled. This filling will cause a large number of redundant calculations, resulting in waste of resources and reducing the efficiency of operations.
- a convolution operation scheme is proposed, which can be executed, for example, by the computing device in FIG. 5 .
- the main processing circuit is used to obtain the input feature map and/or the convolution kernel.
- the input feature map and the convolution kernel have been split into multiple split units according to the convolution split scheme and the dimension storage order is converted, so that a The data in the split unit is stored continuously as a data row.
- the above-mentioned splitting and dimensionality transformation of input feature maps and convolution kernels can be performed at different locations and at different times.
- top_diff can be regarded as the input feature map.
- the main processing circuit may include a block circuit, that is, the block circuit is integrated in the main processing circuit for splitting and dimensionally transforming and storing the input feature map and the convolution kernel respectively.
- the main processing circuit can read the input feature map and convolution kernel in the original storage format from an external storage circuit (such as DDR), and then use the block circuit to split and dimensionally convert the input feature map and convolution kernel respectively, and then One of the input feature map and the convolution kernel is stored in a first storage circuit, and the other is stored in a second storage circuit.
- the above splitting process can be performed during or before the operation to prepare the data.
- the main processing circuit may include a part of the block circuit, which is used to split and dimensionally convert and store only the data determined as multicast data in the input feature map and convolution kernel,
- the data determined to be distributed data can be split and dimensionally converted by an external block circuit.
- the convolution kernel determined to be distributed data can be pre-stored in the second storage circuit after being split and dimensionally converted by an external circuit, which can be directly stored from the off-chip storage circuit to the second storage circuit. In the circuit, it can also be stored in the second storage circuit via the first storage circuit.
- the main processing circuit may not include or perform the function of a blocking circuit at all.
- the input feature maps and convolution kernels are split and dimensionally transformed and stored by a block circuit independent of the main processing circuit.
- One of the split and dimensionally converted input feature map and the convolution kernel may be stored in the first storage circuit, and the other may be stored in the second storage circuit.
- the corresponding convolution splitting scheme can be determined according to the size of the lowest storage dimension (eg Ci) of the input feature map, where the convolution splitting scheme at least indicates the shape of the split unit of the data to be operated.
- the amount of data contained in a split unit does not exceed the maximum single operation amount of the hardware.
- the amount of data contained in a split unit can be set as the one-time processing alignment value M of the hardware, so that the calculation and processing can be performed in units of split units, which can fully utilize the computing power of the hardware and avoid or reduce invalid calculations. .
- the data type can be Int8, Int16, Float16 or Float32
- the split scheme of 64B ⁇ 1 ⁇ 1 shape is called Forward64
- the split scheme of 16B ⁇ 2 ⁇ 2 shape is called Forward16
- the split scheme of 4B ⁇ 4 ⁇ 4 shape is called Forward4
- the split scheme of 4B ⁇ 4 ⁇ 4 shape is called Forward4.
- the split scheme of 4B ⁇ 4 ⁇ 4 shape applied to depth convolution operation is called Forward1
- the split scheme of 4B ⁇ 4 ⁇ 4 shape applied to reverse depth convolution operation is called Update1
- the 4-shape split scheme applied to the cross-product convolution operation is called Update4.
- these splitting schemes are suitable for scenarios where channel C is relatively small in convolution calculations, so they can also be collectively referred to as small convolutions.
- a split unit includes data of the lowest storage dimension and at least one other storage dimension, and the total data volume of a split unit does not exceed the maximum single operation of the hardware.
- the corresponding convolution splitting scheme may be determined according to at least one of the following rules:
- M/4 n can be 64, 16 and 4.
- the size Uc (that is, blockC) of the split unit in the lowest storage dimension is determined to be 4.
- the input feature map and convolution kernel can be split into multiple corresponding splitting units according to the determined convolution splitting scheme, and the dimension storage order can be converted, so that a split
- the data in the unit is continuously stored as a data row, so as to facilitate subsequent reading processing in units of split units (data rows).
- one or more split units may be read in the first reading order from the data to be operated stored in the storage order of the first dimension, in units of split units, and the read split units may be stored in On the corresponding storage circuit, the data in each split unit is stored according to the storage order of the second dimension, and the data between the split units is stored according to the storage order of the third dimension.
- FIG. 6 shows an exemplary data storage sequence according to an embodiment of the present disclosure.
- 610 represents the storage method of the four-dimensional tensor to be calculated, including N three-dimensional sub-tensors, and N is in the highest dimension, that is, the storage order of the first dimension of the four-dimensional tensor is NHWC.
- H and Y, W and X are used interchangeably herein.
- Each subtensor is divided into smaller data blocks or split units, and the number of data blocks in each dimension is C/Y/X respectively.
- the diagram 620 in the middle shows the storage method of each sub-tensor, and each data block is stored as a continuous 64Byte, that is, one row.
- the order between rows changes accordingly.
- the data block is read in the direction of C first, then X, and finally Y, that is, the first reading sequence is YXC, and the rows are stored in the order of Y*X*C, that is, the third
- the dimension storage order is YXC or HWC.
- the third dimension is stored in the same order as the first dimension. It can be understood that other reading orders may also be used, resulting in the storage order of the third dimension being different from that of the first dimension, which will not be listed here.
- the diagram 630 on the right shows the order in each row, that is, the order of data in each data block, and its shape is blockC*blockY*blockX. At this time, the storage order of the second dimension is CYX or CHW.
- the specific splitting scheme will be described in detail later in conjunction with various exemplary convolution splitting schemes.
- the hardware structure of the computing device of the embodiment of the disclosure and the exemplary splitting scheme and storage method of the data are described above.
- the above hardware structure can provide different data paths for the input feature maps and weights involved in the operation, so as to use different data Transmission methods (for example, broadcast, multicast, distribution, etc.) are used to reduce the amount of data access during operation and improve operation efficiency.
- the calculation of convolution is that each input feature map needs to be multiplied and added with each Co convolution kernel to output Co output feature maps.
- the on-chip space can necessarily store convolution kernels and input feature maps of all sizes at the same time. Therefore, for the hardware, there are a series of operations that repeatedly load input feature data or weight data.
- convolution kernel multiplexing can be divided into intra-channel convolution kernel multiplexing and inter-batch convolution kernel multiplexing.
- In-channel convolution kernel multiplexing is for a single output channel, that is, the case of an output feature map, and there is only one set of convolution kernels at this time.
- For each input feature map multiple convolution windows can reuse the same convolution kernel.
- Inter-batch convolution kernel multiplexing is for batch processing, that is, multiple input images are processed simultaneously. Multiple input images are processed with the same set of convolution kernels, so convolution kernels can be reused.
- input feature map multiplexing can be divided into intra-channel input feature map multiplexing and inter-channel input feature map multiplexing.
- Intra-channel input feature map multiplexing is for a single output channel.
- For each input feature map its adjacent convolution windows can reuse part of the input feature map data.
- the multiplexing of input feature maps between channels is for multiple output channels, that is, when there are multiple output feature maps (that is, multiple sets of convolution kernels). At this time, the input feature maps in one convolution window can be convolved with multiple sets The kernel performs the convolution operation.
- the convolution kernel is generally small.
- Kh and Kw are usually single digits
- Co and Ci are about the same size.
- the size of the output channel Co of the convolution kernel in a single round of operation does not exceed the number of scheduled slave processing circuits, so the operation of a single Co needs to be completed by one or more slave processing circuits.
- the dimension of Co is large, it can be realized by splitting into multiple rounds of operations, wherein the size of Co processed by each round of operations does not exceed the number of scheduled slave processing circuits. Therefore, in an example, based on the dimension size of the output channel Co of the convolution kernel and the number of schedulable slave processing circuits Ns, the calculation rounds required to complete the convolution operation and the Co processed in each round of operation can be determined. Quantity or corresponding grouping mode.
- the number of Cos processed by each round may be different, so even for the same Co dimension size, there may be multiple distribution methods.
- it can also be divided into two rounds of calculation.
- the first round processes the first 32 Co values, and each SL processes 2 different Co values; the last round processes the remaining 8 Co values, and every 2
- Each SL handles a different Co value.
- Co 12
- it can be divided into a single round of operation, each SL handles a different Co value, and 4 SLs are idle or perform invalid operations.
- it can also be divided into three rounds of operations, each processing 4 consecutive Co values, and every 4 SLs process a different Co value, so that all schedulable slave processing is utilized in each round of operations circuit. It can be understood that those skilled in the art can also conceive of more other allocation schemes.
- the convolution kernel is multiplexed on Rs SLs in the same SLB, and Rs represents the number of times the convolution kernel is multiplexed between slave processing circuits.
- Factors such as the limitation of hardware buffer space (such as the size of the first buffer circuit and the second buffer circuit in Figure 5) can be considered to determine the maximum number of times rs of convolution kernel multiplexing and the maximum number of input feature map multiplexes applicable in a single slave processing circuit. Use the number of times rn.
- the situation that a slave processing circuit processes multiple Co values in a single round of operation is not considered for the time being, but only one or more slave processing circuits are considered.
- the circuit only handles the case of one Co value in a single round of operation.
- Different grouping modes can be used according to the number of slave processing circuits SL processing the same Co value in a single round of operation. It can be understood that it is preferable to evenly distribute the callable slave processing circuits SL, so as to balance the computing power, for example, every 2 SLs, so that 16 SLs can process 8 Co values at the same time; or every 4 SLs, so that 16 SLs can handle 4 Co values simultaneously; etc.
- the following grouping modes can be selected: Group1 mode, Group4 mode and Group16 mode.
- each grouping mode can refer to the above three representative grouping modes given herein for corresponding processing.
- the above grouping mode can be uniformly expressed as GroupN, representing that all slave processing circuits SL scheduled in the current round of operations are divided into N groups, each slave processing circuit group SLB processes the same Co value, and different slave processing circuit groups SLB handles different Co values.
- N can be 1, 4, or 16, corresponding to Group1, Group4, and Group16 above.
- Figures 7a-7d illustrate several exemplary grouping schemes according to embodiments of the present disclosure.
- Figure 7a shows a Group1 mode
- Figure 7b shows a Group16 mode
- Figure 7c shows a Group4 mode
- Figure 7d shows another Group4 mode.
- the Group1 mode means that all 16 schedulable SLs belong to one group and jointly process one Co value, for example, SL0-SL15 belong to group G0. Thus, operations for this one output channel are distributed over 16 SLs.
- priority can be given to broadcasting the convolution kernel 720 of the output channel to each SL, and the input feature map 710 is split and distributed to each SL, thereby improving memory access efficiency.
- the convolution kernel can be stored in the first storage circuit 530 in FIG. 5 for transmission using a broadcast channel.
- the input feature map can be divided according to the XY direction of the output feature map and stored in the second storage circuit 540 to be allocated to different SLs.
- all SLs jointly compute an output feature map of Co.
- the Group16 mode means that all 16 schedulable SLs are divided into 16 groups, that is, each group has one SL, and each SL handles a different Co value.
- SL0 belongs to group G0
- SL1 belongs to group G1
- SL15 belongs to group G15.
- the same input feature map 730 can be reused among 16 SLs, so it can be prioritized to broadcast the input feature map 730 to each SL, while the convolution kernel 740 corresponding to different Co is distributed Give the corresponding SL.
- the input feature map may be stored in the first storage circuit 530 in FIG. 5 for transmission using a broadcast channel.
- the convolution kernels are divided according to Co and stored in the second storage circuit 540 to be allocated to different SLs.
- all SLs compute output feature maps of different Co for the same input feature map.
- Group4 mode all 16 schedulable SLs are divided into 4 groups, and each group processes a Co value.
- SL0-SL3 belong to group G0
- SL4-SL7 belong to group G1
- SL8-SL11 belong to group G2
- SL12-SL15 belong to group G3.
- This mode is between Group1 and Group16, so either the convolution kernel or the input feature map can be determined as multicast data, while the other can be determined as distribution data.
- the convolution kernels can be divided into 4 groups according to Co, and stored in the first storage circuit 530 in FIG. 5 , so as to be transmitted through a broadcast channel.
- the input feature map can be divided into 4 parts according to the XY direction of the output feature map, copied into 4 parts, stored in the second storage circuit 540, and distributed to the 4 SLBs.
- Each SLB obtains the same input feature map, and then distributes it to the 4 SLs in the SLB according to the 4 divided parts.
- all SLs in each SLB jointly compute the output feature map of a Co, and the 4 SLBs process a different Co respectively.
- the convolution kernel can be stored in the second storage circuit 540 in FIG. 5
- the input feature map can be stored in the first storage circuit 530
- the division method is similar to the previous embodiment.
- FIG. 7c shows a Co allocation manner 770 of a convolution kernel.
- the convolution kernels are divided into 4 groups, and each group is divided into each group at an interval of 1 according to Co.
- Co 12
- four groups of Co are divided into ⁇ 0, 4, 8 ⁇ , ⁇ 1, 5, 9 ⁇ , ⁇ 2, 6, 10 ⁇ and ⁇ 3, 7, 11 ⁇ respectively.
- Fig. 7d shows another way of Co allocation 780 of the convolution kernel.
- the input feature map needs to be split between these multiple SLs.
- the Group1 grouping mode needs to split the input feature map into 16 parts.
- the Group4 grouping mode needs to split the input feature map into 4 parts.
- the input feature map may be divided among the Rs slave processing circuits SL included in each slave processing circuit group as follows: according to the size of the corresponding output feature map, the output feature map is divided in the XY dimension (also That is, the Ho/Wo dimension) is evenly divided into Rs output feature blocks of the same shape; and according to the input feature map area required for calculating each output feature block, the input feature map is divided in the XY dimension (that is, the Hi/Wi dimension) The above is divided into Rs input feature blocks to be distributed to Rs slave processing circuits. It can be understood that depending on the size of the convolution kernel and the convolution step size, the input feature maps corresponding to adjacent output points on the output feature map may overlap.
- Fig. 8 shows an exemplary split diagram of an input feature map according to an embodiment of the present disclosure.
- the input feature map is divided into 16 parts and distributed on 16 SLs, corresponding to the Group1 mode.
- the 16 output feature blocks can be mapped to the input feature map 820 to obtain the 16 input feature map regions required to calculate the 16 output feature blocks respectively, which also divides the input feature map in the XY direction.
- These 16 input feature map regions can be assigned to 16 slave processing circuits SL accordingly.
- the input feature map will be split in units of splitting units according to the determined convolution splitting scheme. Therefore, in the above embodiment, the block of the input feature map should make each divided input feature map
- the block in the XY direction is a multiple of the dimension of the split unit in the XY direction, that is, it can be aligned according to the split unit in the XY direction. For example, when choosing a 4 ⁇ 4 ⁇ 4 convolution splitting scheme, each input feature map is aligned by 4 ⁇ 4; while choosing a 16 ⁇ 2 ⁇ 2 convolution splitting scheme, each input feature map Blocks are aligned 2 ⁇ 2.
- the output feature map is not aligned according to the split unit (such as 4 ⁇ 4 or 2 ⁇ 2)
- it is necessary to fill in the input feature map accordingly (such as filling 0), so that the actual calculated output XY is according to the split unit ( eg 4x4 or 2x2) aligned and input XY is also aligned by split unit (eg 4x4 or 2x2).
- the output feature map can also be divided according to other rules in the XY direction, for example, divided into 16 output feature blocks with the same shape according to 1 ⁇ 16, and assigned to SL0-SL15 respectively.
- Embodiments of the present disclosure are not limited in this respect.
- this splitting method can also be applied to splitting in other scenarios, for example, between computing circuits CU in a single slave processing circuit SL Splitting, the embodiments of the present disclosure are not limited in this aspect.
- one of the input feature map or the convolution kernel can be stored in the first storage circuit 530 in FIG. 5 , and the other can be stored in the second storage circuit 540 .
- Data in the first storage circuit may be multicast via a broadcast path, while data in the second storage circuit is typically distributed. By reasonably allocating the storage methods of each data, the speed of data access can be accelerated.
- the second storage circuit may allocate a storage area to each slave processing circuit SL, so that the data required for operation of each slave processing circuit only needs to be read from its corresponding storage area.
- FIGS. 9a-9d show schematic diagrams of data storage in the second storage circuit according to an embodiment of the present disclosure.
- Each storage area stores the convolution kernel or input feature map to be processed by the slave processing circuit. It can be understood that depending on different grouping modes, the storage content in each storage area will also be different.
- Fig. 9a shows that in Group1 mode, the input feature map is split into 16 parts FB0-FB15 and stored in each storage area of the second storage circuit.
- a continuous two-dimensional area is stored in the storage area corresponding to each SL, and these two-dimensional areas are divided according to, for example, the manner shown in FIG. 8 .
- the split units described above are stored in rows, that is, one row corresponds to a split unit of an input feature map. For example, assuming that each input feature block after splitting includes 4 split units, that is, 4 lines of data, then in the storage area 1100 allocated to SL0, the first line Line01, the second line Line02, The third line Line03 and the fourth line Line04 input feature maps. Each row can also be called an input feature row.
- FIG. 9b shows that in the Group16 mode, the convolution kernels are divided according to Co and stored in each storage area of the second storage circuit to be allocated to corresponding SLs. Convolution kernels assigned to different Co values are stored in the storage area corresponding to each SL. For example, two Co allocation methods are described above, and correspondingly, there are two storage methods. One of them is shown in FIG. 9 b , that is, in each round of operation, consecutive Co values are assigned to each SL in sequence. In this way, after each round of calculation is completed, the Co dimensions of the calculation results output by each SL are continuous.
- the input feature map may also be stored in the second storage circuit (not shown). At this time, the input feature map does not need to be split, and is directly copied into 16 copies, which are stored in each storage area of the second storage circuit to be allocated to the corresponding SL, so that each SL can target the same input feature map and different Co values
- the convolution kernel performs the convolution operation.
- Fig. 9c shows a possible storage content in the Group4 mode.
- the input feature map is split into 4 parts and copied into 4 parts, and stored in each storage area of the second storage circuit.
- each slave processing circuit group SLB processes the same input feature map and different Co convolution kernels; and the four SLs in each SLB respectively process a split input feature map block. Therefore, the storage contents of the storage areas used for the four SLBs in the figure are the same, for example, the contents in 900-903 are the same as the contents in 912-915.
- the storage areas for different SLs store different split input feature blocks, for example, the input feature block FB0 is stored in 900, the input feature block FB1 is stored in 901, and so on.
- the same storage allocation is also performed in the storage areas of other SLBs, which will not be repeated here.
- Fig. 9d shows another possible storage content in the Group4 mode.
- the convolution kernels are divided into 4 groups according to Co, and stored in each storage area of the second storage circuit.
- the convolution kernels are divided into groups with an interval of 1 according to Co.
- Co 16
- the 4 SLs in each SLB share the same weight.
- the same weights are stored in storage areas 900 , 901 , 902 and 903 .
- Co can also adopt a continuous manner within a single SLB, and those skilled in the art can deduce its storage manner by referring to the foregoing description, which will not be described in detail here.
- multiple slave processing circuits can be scheduled to perform convolution operations on the input feature map and the corresponding data rows of the convolution kernel, and then according to the convolution splitting scheme, A plurality of operation results returned from the processing circuit are spliced to obtain an output feature map of the convolution operation of the input feature map and the convolution kernel.
- a plurality of operation circuits CU and each buffer circuit (see FIG. 5 ) in the slave processing circuit can be used to perform a specific convolution operation process.
- the first buffer circuit can be used to cache the input feature map, which may come from the first storage circuit or the second storage circuit; correspondingly, the second buffer circuit can be used to cache the convolution kernel, which may come from the first The second storage circuit or the first storage circuit.
- each operation circuit CU can perform each calculation for the data row selected from the first buffer circuit (for example, the input feature row) and the data row selected from the second buffer circuit (for example, the weight value row) Performs a bitwise multiply-accumulate operation.
- the following description focuses on the processing in a single slave processing circuit SL, and it can be understood that similar processing is performed in other SLs.
- each output feature block corresponds to all schedulable N CU operation circuits in a single SL A single calculation capability (N CU *Nop output points).
- the output feature map can be divided into output feature blocks according to the alignment of 16 output points in the XoYo dimension, and each output feature block can be calculated one by one. It can be understood that the 16 output points may be in a 4*4 format, or may be in a 1*16 format, which is not limited in the embodiment of the present disclosure.
- the output points of the output characteristic block can be further divided among the N CU operation circuits, so as to determine the processing object of each operation circuit. Then, according to the division of output points, using the split unit as a sliding window, select N CU input feature data rows from the first buffer circuit and distribute them to N CU computing circuits, and select the corresponding weight value from the second buffer circuit The data is broadcast to N CU computing circuits, so that the parallel calculation of the output points corresponding to multiple sliding windows can be realized by multiplexing the weight data. Perform Nk sliding selections, wherein Nk is determined according to the smaller value of the size of the convolution kernel in the X and Y dimensions and the maximum convolution kernel size supported by a single operation of the processing circuit.
- the corresponding weight data when performing a three-dimensional convolution operation, can be selected as follows: select 1/Nop weight rows from the second buffer circuit in a sliding manner corresponding to that in the first buffer circuit , expand its copy Nop-1 into an extended weight row, and broadcast it to N CU computing circuits in the slave processing circuit.
- each operation circuit can perform the operation in units of 1/Nop data lines for one input feature line from the first buffer circuit and one extended weight data line from the second buffer circuit during each sliding number selection period.
- bitwise multiplication and accumulation Nop partial sums are obtained; and the Nk*Nop partial sums calculated during the Nk sliding number selection period are accumulated according to the corresponding convolution output points, and Nop operation results are obtained and output.
- the slave processing circuit When the slave processing circuit outputs the output points of its internal operation circuit, it can output the output points calculated by multiple operation circuits in it in a specific order according to the division method of the output points, so that the output points of continuous output are in X and/or Y Dimensionally continuous, convenient for subsequent processing.
- the aforementioned block circuit may further store the operation results returned from each slave processing circuit in a fourth-dimensional storage order. According to the situation, the block circuit can also convert the operation result into the desired dimension storage order for storage.
- 10a-10b show schematic diagrams of two different divisions of output points between operational circuits.
- Fig. 10a shows a schematic diagram of assigning continuous output points to each arithmetic circuit according to some embodiments of the present disclosure.
- the output feature block can be equally divided into N CU output feature sub-blocks with the same shape among N CU operation circuits, and each output feature sub-block includes Nop output points, so that each operation
- the circuits are respectively responsible for computing one of the output feature sub-blocks.
- the output feature block 1010a includes 4*4 output points
- each operation circuit Compute consecutive 2*2 output points (or partial sums) each time.
- different backgrounds are used to show the output points assigned to four different arithmetic circuits CU0-CU3.
- the data required for calculating the output feature sub-blocks can be selected from the first buffer circuit corresponding to the positions of the N CU output feature sub-blocks.
- N CU data rows are used for operation.
- the first input data row can be selected from the corresponding input feature blocks and distributed to 4 arithmetic circuits.
- the corresponding weight data can be selected from the second buffer circuit and broadcast to NCU computing circuits, so as to achieve parallel calculation of output points corresponding to multiple computing circuits by multiplexing the weight data .
- weight multiplexing can be performed in a single input data row , thus computing Nop output points or partial sums simultaneously.
- the extended weight value row can also be broadcast to N CU computing circuits, so that while multiplexing the weights among multiple computing circuits, a smaller granularity (for example, 1 /Nop line) to reuse weights.
- N CU *Nop output points or partial sums can be calculated each time by correspondingly taking N CU input feature data rows and taking 1/Nop weight value rows to copy and expand into 1 weight value row.
- the calculation result is a partial sum
- the partial sum can be calculated multiple times by sliding multiple times, and the partial sums of each time are accumulated according to the output points to which they belong, and the final result can be obtained.
- the number of slides and the slide step of the convolution operation can be determined.
- the maximum convolution kernel size supported by a single operation of the processing circuit is, for example, determined by the space sizes of the first buffer circuit and the second buffer circuit. It can be understood that when the convolution kernel exceeds the maximum convolution kernel size, it needs to be split in the Kx and Ky directions according to the maximum convolution kernel size.
- the operation results of each operation circuit can be output one by one.
- the operation results of one operation circuit are output each time, for example, 2*2 output points, and 4*4 output feature blocks are returned for 4 consecutive times.
- Fig. 10b shows a schematic diagram of assigning interval output points to each operation circuit according to other embodiments of the present disclosure.
- the output feature block can be equally divided into Nop output feature sub-blocks with the same shape among N CU computing circuits, each output feature sub-block includes N CU output points, and is divided into N CU output points respectively.
- CU arithmetic circuits For example, the figure still takes the above example as an example, showing that the output feature block 1010b includes 4*4 output points, and each output feature sub-block 1011b-1011b divided equally includes 2*2 output points. In each output feature sub-block, the 2*2 output points are allocated to 4 operation circuits. Thus, each operation circuit calculates one output point in each of the Nop output feature sub-blocks. In the figure, different backgrounds are used to show the output points assigned to four different arithmetic circuits CU0-CU3.
- the output point position of each output feature sub-block can be correspondingly obtained from the first buffer circuit according to the data required for calculating the output feature sub-block. Select N CU data rows for operation.
- the interval or step size of the four input data rows selected at the same time in the X and/or Y direction is 1.
- the corresponding weight data can be selected from the second buffer circuit and broadcast to NCU computing circuits, so as to achieve parallel calculation of output points corresponding to multiple computing circuits by multiplexing the weight data .
- weight multiplexing can be performed in a single input data row , thus computing Nop output points or partial sums simultaneously.
- the extended weight value row can also be broadcast to N CU computing circuits, so that while multiplexing the weights among multiple computing circuits, a smaller granularity (for example, 1 /Nop line) to reuse weights.
- N CU *Nop output points or partial sums can be calculated each time by correspondingly taking N CU input feature data rows and taking 1/Nop weight value rows to copy and expand into 1 weight value row.
- the calculation result is a partial sum
- the partial sum can be calculated multiple times by sliding multiple times, and the partial sums of each time are accumulated according to the output points to which they belong, and the final result can be obtained.
- the number of slides and the slide step of the convolution operation can be determined.
- the number of slides Nk ceil(Kx/2)*ceil(Ky/2), where Kx and Ky are the dimensions of the convolution kernel in the X and Y dimensions and are supported by a single operation from the processing circuit
- Kx and Ky are the dimensions of the convolution kernel in the X and Y dimensions and are supported by a single operation from the processing circuit
- the smaller value of the maximum convolution kernel size of , sliding step 2.
- the maximum convolution kernel size supported by a single operation of the processing circuit is determined, for example, by the space sizes of the first buffer circuit and the second buffer circuit. It can be understood that when the convolution kernel exceeds the maximum convolution kernel size, it needs to be split in the Kx and Ky directions according to the maximum convolution kernel size.
- the output points calculated by each operation circuit are spaced in the X and/or Y dimensions, that is, discontinuous, it is necessary to select part of the operation results of some operation circuits for output each time. so that the output points are continuous in the X and/or Y dimensions.
- the first line needs to output two results from CU0 and two results from CU1
- the second line needs to output two results from CU2 and two results from CU3, and so on.
- the first operation result of each of CU0-CU3 is output for the first time
- the second operation result of each of CU0-CU3 is output for the second time
- the operation result may also be output in columns, which will not be repeated here.
- a slave processing circuit can calculate multiple 4*4 areas in the Xo/Yo direction, for example, calculate up to 16 4*4 areas. At this time, weights or neurons can be reused according to the storage content in the second storage circuit, and the reading frequency of the second storage circuit can be reduced. If the calculated result is a partial sum, it is stored in a register in the arithmetic circuit.
- each slave processing circuit can control the reading mode of the weight value data line and the input feature map data line according to the weight value multiplexing and/or input feature map multiplexing mode, so as to combine the weights through multiple operations.
- the value data and the input feature map data traverse the entire convolution window of the convolution output point at the same time to perform the bitwise multiplication and accumulation operation, and obtain multiple partial sum results and accumulate them to obtain the convolution output on the corresponding convolution output point.
- the shape of the split unit is 16B ⁇ 2 ⁇ 2, and its operation process can also be applied to a similar convolution split scheme.
- Ci the input feature map
- Ci the Ci dimension of the convolution kernel
- the split unit can be 32B ⁇ 2 ⁇ 2, 8B ⁇ 4 ⁇ 4 shapes.
- the convolution splitting scheme it is necessary to align the input feature map and the Ci dimension of the convolution kernel to 32B or 8B.
- Fig. 11 shows a schematic diagram of splitting and storage of the Forward16 scheme according to Embodiment 1 of the present disclosure.
- the example in the figure assumes the data type is Int8.
- the Forward16 convolution splitting scheme can be implemented by the block circuit integrated in the main processing circuit, or the block completely or partially independent of the main processing circuit
- the circuit splits the input feature map and convolution kernel into multiple corresponding split units.
- the block circuit can also convert the dimension storage order of the input feature map and the convolution kernel, so that the data in each split unit is continuously stored as a data row.
- the split and transformed input feature maps and/or convolution kernels may be provided to a master processing circuit or a slave processing circuit.
- the main processing circuit can distribute the data obtained by it to multiple slave processing circuits for performing convolution operations;
- the output feature map of the convolution operation of the feature map and the convolution kernel can perform convolution operations based on the data they obtain, and return the operation results to the main processing circuit.
- the convolution splitting scheme can also indicate the number of operation rounds L required to perform the convolution operation, where the number of output channels Co processed in each operation round corresponds to the number of slaves that can be scheduled in the operation round.
- the number of processing circuits is Ns, so that one value of Co can be processed by one slave processing circuit.
- the input feature map can be multiplexed between these slave processing circuits.
- the input feature map can be determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so as to be transmitted through the broadcast bus during operation to the scheduled multiple slave processing circuits.
- the convolution kernel may be determined as distribution data, and the distribution data after splitting and converting the dimension storage sequence is stored in the second storage circuit, so as to be distributed to corresponding slave processing circuits. These distributed data can be distributed to corresponding slave processing circuits before operation.
- convolution kernels with different Co values assigned to each slave processing circuit in each operation round may be further stored in storage areas allocated to corresponding slave processing circuits in the second storage circuit.
- storage content in the second storage circuit for example, refer to FIG. 9b.
- the first buffer circuit can buffer a plurality of input feature data rows from the first storage circuit and broadcast transmission; and the second buffer circuit can buffer the convolution kernel distributed to the slave processing circuit from the second storage circuit Multiple weight data rows for .
- these data rows may be distributed to corresponding computing circuits or broadcast to all computing circuits in the slave processing circuit during computing.
- each operation circuit CU can perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the weight data row selected from the second buffer circuit in each operation.
- the splitting method of the output points between the four computing circuits CU can refer to Figure 10a, for example, that is, at each calculation, each computing circuit calculates output feature maps that are continuous in the X and/or Y dimensions multiple output points.
- Fig. 12 shows a schematic diagram of a single operation process in the Forward16 scheme according to an embodiment of the present disclosure.
- the size of the first buffer circuit 1210 is 3 ⁇ 3 ⁇ 64B, that is, a maximum of 9 rows of data can be buffered
- the size of the second buffer circuit 1220 is 2 ⁇ 2 ⁇ 64B, that is, a maximum of 4 rows of data can be buffered .
- the storage in the buffer circuit in the figure is also shown in the split unit.
- the figure shows the operation process of the first sliding fetch.
- using the split unit as a sliding window slidingly select N CU input feature lines from the first buffer circuit, and send them to N CU computing circuits for calculation; from the second buffer circuit
- 1/Nop weight rows are selected according to the corresponding sliding method in the first buffer circuit, where Nop is the maximum number of convolution output points that can be calculated for each operation circuit at a time, and its copy Nop-1 is expanded to An extended weight row is broadcast to N CU computing circuits in the slave processing circuit.
- each operation circuit calculates the output feature blocks including 2 ⁇ 2 output points for each calculation.
- one input characteristic data row is selected from the four input characteristic blocks corresponding to the divided output points at the initial position in the first buffer circuit 1210 and correspondingly sent to the four input characteristic blocks in the slave processing circuit SL.
- Operation circuit 1240 Select 1/4 weight data line at the starting position from the second buffer circuit 1220, copy 3 copies of it and expand it into an extended weight data line 1230, and broadcast it to the 4 arithmetic circuits 1240 in the SL.
- each operation circuit performs bitwise multiplication and accumulation in units of 1/Nop data lines for one input feature row from the first buffer circuit and one extended weight value row from the second buffer circuit to obtain Nop parts and.
- the four computing circuits 1240 perform a bitwise multiplication and accumulation operation on the distributed input feature data row and the broadcasted extended weight data row to obtain the computing result 1250.
- the results of different background colors in 1250 represent the results obtained by different computing circuits 1240. owned. It can be seen that for each calculation, one CU will calculate one 2 ⁇ 2 partial sum, and four CUs will obtain four 2 ⁇ 2 partial sums in total, that is, 4 ⁇ 4.
- the number is slidingly fetched synchronously, and the next calculation is performed.
- the operation circuit accumulates the Nk*Nop partial sums calculated during the Nk sliding calculations according to the corresponding convolution output points to obtain and output Nop operation results.
- the maximum convolution kernel size supported by a single operation of the slave processing circuit is 3 ⁇ 3.
- Fig. 13 shows a schematic diagram of a sliding convolution process in the Forward16 scheme according to an embodiment of the present disclosure.
- This example takes an input feature map of 6 ⁇ 6 and a convolution kernel of 3 ⁇ 3 as an example.
- the convolution step size is 1, and the output feature map size is 4 ⁇ 4.
- the input feature map has been aligned to 2 ⁇ 2, divided into 9 blocks of 16 ⁇ 2 ⁇ 2 (C ⁇ H ⁇ W) size, and stored in the first buffer circuit, shown as 1310 in the figure, where the C dimension is omitted .
- the convolution kernel 3 ⁇ 3 needs to be aligned to 4 ⁇ 4, and the aligned part is filled with 0, and stored in the second buffer circuit, which is shown as 1320 in the figure, and the C dimension is also omitted.
- the copy operation can be realized by hardware.
- FIG. 13 The input feature map and the selection range of the convolution kernel in the first buffer circuit and the second buffer circuit for each slide are shown in Figure 13.
- block 1310 represents the input feature map in the first buffer circuit, and the four dotted-line boxes represent the areas selected to be sent to four CUs;
- block 1320 represents the convolution kernel in the second buffer circuit, and the dotted-line boxes represent the selected 1/ 4 lines, which are copied into 3 copies and expanded into one line and then broadcast to 4 CUs.
- each CU performs bitwise multiplication and accumulation in units of 1/4 data line for one data line from the first buffer circuit and one extended data line from the second buffer circuit to obtain 4 partial sums ; and accumulating the Nk partial sums corresponding to the same convolution output point calculated for Nk times in the current operation round, to obtain and output 4 operation results.
- each CU calculates the partial sum of the 4 output points on the output feature map, the partial sum is the alignment of 1/4 data row
- Fig. 14 shows a schematic diagram of accumulation of sliding convolution results in the Forward16 scheme according to an embodiment of the present disclosure.
- each operation circuit CU uses 1/4 data for one input feature data line from the first buffer circuit and one extended weight data line from the second buffer circuit during each calculation.
- the row unit is multiplied and accumulated to obtain 4 partial sums.
- each slave processing circuit SL can convert the operation result of its internal operation circuit CU into a specified format, for example, the format of Nco ⁇ Uy ⁇ Ux.
- each slave processing circuit may output Nop operation results of one operation circuit within it each time in the order of continuous division of output points.
- the block circuit may further store the operation results returned from each slave processing circuit in a fourth dimension storage order. According to the situation, the block circuit can also convert the operation result into the desired dimension storage order for storage.
- Fig. 15 shows a schematic diagram of the output data format of the Forward16 splitting scheme according to an embodiment of the present disclosure.
- each CU 1510 in the figure shows the raw output of 1 SL.
- each CU computes 2 ⁇ 2 output neurons. Since the four output neurons calculated by one CU are adjacent, each SL can output the calculation result of one of the CUs each time in the order of continuous division of output points, that is, 1 ⁇ 2 ⁇ 2(Co ⁇ The area of Y ⁇ X) returns the area of 1 ⁇ 4 ⁇ 4 (Co ⁇ Y ⁇ X) for 4 consecutive times, that is, the 4 calculation results of each of the 4 CUs.
- Different CUs within the same SL output different regions of the output feature map of the same Co. Different SLs output different Co output feature maps.
- the output buffer circuit (such as the third buffer circuit in FIG. 5 ) can convert the output result into a 16 ⁇ 2 ⁇ 2 format, where 16 corresponds to the number of SLs and also corresponds to the number of output channels Co.
- a single slave processing circuit including four operation circuits can calculate up to 16 output feature regions of 4 ⁇ 4, so the weights can be reused, thereby reducing the 2.
- the reading frequency of the storage circuit That is, the reading frequency of the first storage circuit and the second storage circuit may be different. If the result calculated by the arithmetic circuit is a partial sum, it is stored in the internal register.
- the slave processing circuit can be further used to: determine the weight multiplexing times rs in the slave processing circuit according to the storage space limitation in the arithmetic circuit; and control the loading frequency of the input feature data in the first buffer circuit , so that the weight data loaded each time in the second buffer circuit is reused rs times, and performs a convolution operation with the corresponding input feature data loaded rs times in the first buffer circuit.
- rs may take a value no greater than 16.
- the shape of the split unit is 4B ⁇ 4 ⁇ 4, and its operation process can also be applied to a similar convolution split scheme.
- Ci the input feature map
- Ci the Ci dimension of the convolution kernel
- Fig. 16 shows a schematic diagram of splitting and storage of the Forward4 scheme according to an embodiment of the present disclosure.
- the example in the figure assumes the data type is Int8.
- 1610 in the figure shows the original data to be operated (which may be neurons or weights), and its storage order is HWC.
- the Forward4 convolution splitting scheme can be implemented by the block circuit integrated in the main processing circuit, or the block completely or partially independent of the main processing circuit
- the circuit splits the input feature map and convolution kernel into multiple corresponding split units.
- the block circuit can also convert the dimension storage order of the input feature map and the convolution kernel, so that the data in each split unit is continuously stored as a data row.
- the split and transformed input feature maps and/or convolution kernels may be provided to a master processing circuit or a slave processing circuit.
- the main processing circuit can distribute the data obtained by it to multiple slave processing circuits for performing convolution operations;
- the output feature map of the convolution operation of the feature map and the convolution kernel can perform convolution operations based on the data they obtain, and return the operation results to the main processing circuit.
- the convolution kernel when the number of input channels is small, the convolution kernel is generally small. For example, Kh and Kw are usually single digits, and Co and Ci are about the same size. In these embodiments, usually the size of the output channel Co of the convolution kernel in a single round of operation does not exceed the number of scheduled slave processing circuits, so the operation of a single Co needs to be completed by one or more slave processing circuits. More generally, even if the dimension of Co is large, it can be realized by splitting into multiple rounds of operations, wherein the size of Co processed by each round of operations does not exceed the number of scheduled slave processing circuits.
- the calculation rounds required to complete the convolution operation and the Co processed in each round of operation can be determined. Quantity or corresponding grouping mode.
- the Forward4 convolution splitting scheme can support the three grouping modes described above in conjunction with Figure 7: Group1, Group4, and Group16.
- the convolution kernel can be determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so that in During the operation, it is transmitted to the scheduled multiple slave processing circuits through the broadcast bus.
- the input feature map may be determined as the distribution data, and the distribution data after splitting and transforming the dimension storage sequence is stored in the second storage circuit, so as to be distributed to the corresponding slave processing circuit. These distributed data can be distributed to corresponding slave processing circuits before operation.
- the input feature map can be split among multiple SLs of a single SLB with reference to the schematic diagram of FIG. 8 .
- FIG. 9a Group1
- FIG. 9c Group4
- the first buffer circuit can buffer a plurality of input characteristic data rows from the second storage circuit distributed to the slave processing circuit; and the second buffer circuit can buffer the multicast transmission from the first storage circuit to the slave processing circuit.
- a plurality of weight data rows of the convolution kernel corresponding to the output channel value of the processing circuit.
- these data rows can be distributed to the corresponding computing circuit CU or broadcast to all CUs in the slave processing circuit during the computing period.
- each operation circuit CU can perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the weight data row selected from the second buffer circuit in each operation.
- the splitting method of the output points between the four computing circuits CU can refer to Figure 10b, for example, that is, at each calculation, each computing circuit calculates the output feature map in the X and/or Y dimension interval multiple output points.
- FIG. 17 shows a schematic diagram of a single operation process in the Forward4 scheme according to an embodiment of the present disclosure.
- the size of the first buffer circuit 1710 is 3 ⁇ 3 ⁇ 64B, that is, a maximum of 9 rows of data can be buffered
- the size of the second buffer circuit 1720 is 2 ⁇ 2 ⁇ 64B, that is, a maximum of 4 rows of data can be buffered .
- the storage in the buffer circuit in the figure is also shown in the split unit.
- the figure shows the operation process of the first sliding fetch.
- using the split unit as a sliding window slidingly select N CU input feature lines from the first buffer circuit, and send them to N CU computing circuits for calculation; from the second buffer circuit
- 1/Nop weight rows are selected according to the corresponding sliding method in the first buffer circuit, where Nop is the maximum number of convolution output points that can be calculated for each operation circuit at a time, and its copy Nop-1 is expanded to An extended weight row is broadcast to N CU computing circuits in the slave processing circuit.
- each operation circuit calculates 2 ⁇ 2 output points with an interval of 1 in the X and Y dimensions for division.
- one input feature data line is selected from the first buffer circuit 1710 at the initial position and the position moved by 1 in the X and/or Y directions, and a total of four input feature data lines are selected, and correspondingly sent to the slave Four arithmetic circuits 1740 in the processing circuit SL.
- Select 1/4 weight data row at the starting position from the second buffer circuit 1720 that is, select data of 2 ⁇ 2 size, copy 3 copies of it and expand it into an extended weight data row 1730, and broadcast it to the SL 4 arithmetic circuits 1740 inside.
- each operation circuit performs bitwise multiplication and accumulation in units of 1/Nop data lines for one input feature row from the first buffer circuit and one extended weight value row from the second buffer circuit to obtain Nop parts and.
- four computing circuits 1740 perform bitwise multiplication and accumulation operations on the distributed input feature data row and the broadcasted extended weight data row to obtain the computing result 1750.
- the results of different background colors in 1750 represent the results obtained by different computing circuits 1740. owned. It can be seen that for each operation, one CU will calculate the partial sum of 4 output points, and the 4 CUs will obtain a total of 4 ⁇ 4 partial sums. It can be seen that the output points calculated by each CU are not adjacent in the XoYo dimension of the output feature map.
- the number is slidingly fetched synchronously, and the next calculation is performed.
- Nk ceil(Kx/2)*ceil(Ky/2)
- Kx and Ky are the dimensions of the convolution kernel in the X and Y dimensions respectively or from the processing circuit in the current convolution split mode The smaller value among the maximum convolution kernel sizes supported by a single operation.
- the operation circuit accumulates the Nk*Nop partial sums calculated during the Nk sliding calculations according to the corresponding convolution output points to obtain Nop operation results.
- the maximum convolution kernel size supported by a single operation of the processing circuit is 8 ⁇ 8.
- Fig. 18 shows a schematic diagram of a sliding convolution process in the Forward4 scheme according to an embodiment of the present disclosure.
- This example takes a 9 ⁇ 9 input feature map and a 5 ⁇ 5 convolution kernel as an example. If the convolution step is 1, the output feature map size is 5 ⁇ 5.
- the input feature map needs to be aligned to 12 ⁇ 12, divided into 9 blocks of 4 ⁇ 4 ⁇ 4 (C ⁇ H ⁇ W) size, and stored in the first buffer circuit, shown as 1810 in the figure, where the C dimension is omitted .
- the convolution kernel 5 ⁇ 5 needs to be aligned to 8 ⁇ 8, and the aligned part is filled with 0, and stored in the second buffer circuit, which is shown as 1820 in the figure, and the C dimension is also omitted.
- the copy operation can be realized by hardware.
- FIG. 18 The input feature map and the selection range of the convolution kernel in the first buffer circuit and the second buffer circuit for each sliding are shown in Figure 18, a total of 9 images, representing a total of 9 sliding times.
- block 1810 represents the input feature map in the first buffer circuit, and the four dotted-line boxes represent the areas selected to be sent to the four CUs;
- block 1820 represents the convolution kernel in the second buffer circuit, and the dotted-line boxes represent the selected 1/ 4 lines, which are copied into 3 copies and expanded into one line and then broadcast to 4 CUs.
- each CU performs bitwise multiplication and accumulation in units of 1/4 data line for one input feature data line from the first buffer circuit and one extended weight data line from the second buffer circuit, to obtain 4 partial sums; and accumulating the Nk partial sums corresponding to the same convolution output point obtained in the Nk operation cycles in the current operation round, to obtain and output 4 operation results.
- the cumulative result that is, each output point is a standard convolution of 4 ⁇ 2 ⁇ 2 (Ci ⁇ Y ⁇ X).
- the accumulation is completed in the Y ⁇ X direction
- a complete 4 ⁇ 4 (Y ⁇ X) output is obtained in one SL (as shown in Figure 10b shown).
- a single calculation only supports the case where the convolution kernel is not larger than 8 ⁇ 8.
- each slave processing circuit SL can convert the operation result of its internal operation circuit CU into a specified format, for example, the format of Nco ⁇ Uy ⁇ Ux.
- each slave processing circuit may output a partial operation result of its internal partial operation circuit each time, and the partial operation result is continuous on the X and/or Y dimension of the output feature map.
- the block circuit may further store the operation results returned from each slave processing circuit in a fourth dimension storage order. According to the situation, the block circuit can also convert the operation result into the desired dimension storage order for storage.
- the output data format is slightly different.
- FIG. 19 shows a schematic diagram of an output data format of the Forward4 scheme according to an embodiment of the present disclosure.
- the grouping mode is Group1
- each SL outputs a 1 ⁇ 1 ⁇ 4 (Co ⁇ Y ⁇ X) area each time, that is, it outputs part of the operation results of its internal partial operation circuit each time, for example, each of the 2 CUs has 2 operation results (see FIG. 10 b ), this part of the operation results is continuous on the X and/or Y dimensions of the output feature map, for example, the same row (as shown in FIG. 19 ) or the same column.
- the 1 ⁇ 4 ⁇ 4 (Co ⁇ Y ⁇ X) area is returned 4 times in a row, that is, the 4 operation results of each of the 4 CUs.
- Different SLs output different regions of the output feature map of the same Co. After outputting all the 4 ⁇ 4 areas of Co, continuing to output will switch different output points.
- the 1920 in the figure shows the deposit-out data structure of 16 SLs.
- the final output data becomes the format of Yo*Xo*Co*4*16*4 after being written into the storage circuit (for example, the first storage circuit), where Yo and Xo are the outputs divided by each SL
- the number of blocks in the feature map, 16 is the division on 16 SLs.
- pendulum operations can be performed again to convert to other desired data formats.
- the output data format is also slightly different. Assuming the original output size is:
- (4*16*4) is the basic output block of forward4, and the directions correspond to h*c*w respectively, where 16 represents the division of ho and wo of the same co on 16 SLs; ho and wo are divided twice 4, where the first 4 indicates 4 ⁇ 4 splitting when storing data in SL, and the second 4 indicates data block folding in h and w directions.
- This shape is also the shape of the schematic diagram in FIG. 19 .
- the Group4 output data shape is:
- (4*16*4) has the same meaning as above, except that 16 represents the wo output division of 4 cos on 4 SLs.
- the Group16 output data shape is:
- (4*16*4) has the same meaning as above, except that 16 represents the output division of 16 COs on 16 SLs.
- the hardware when outputting, can automatically output neurons according to the dimension of 4*16*4(Y*SL*X) in the row and the dimension of Y*X*C between the rows. The same is true for larger convolution kernels.
- a single slave processing circuit including four operation circuits can calculate up to 16 4 ⁇ 4 output feature regions, so the input feature map/neuron can be multiplexed , thereby reducing the reading frequency of the second storage circuit. That is, the reading frequency of the first storage circuit and the second storage circuit may be different. If the result calculated by the arithmetic circuit is a partial sum, it is stored in the internal register.
- the slave processing circuit can be further used to: determine the input feature multiplexing times rn in the slave processing circuit according to the storage space limitation in the arithmetic circuit; and control the loading frequency of the weight data in the second buffer circuit , so that the input feature data loaded each time in the first buffer circuit is reused rn times, and the convolution operation is performed with the corresponding weight data loaded in the second buffer circuit rn times.
- rn may take a value no greater than 16.
- Forward1 the shape of the split unit is the same as Forward4, which is also 4B ⁇ 4 ⁇ 4; the difference is that Forward1 is applied to depth convolution operations, that is, 2D convolution operations.
- depth convolution operations that is, 2D convolution operations.
- 2D convolution operations For the principle of the 2D convolution operation, reference may be made to the previous description in conjunction with FIG. 4b. The following description can be applied in a convolution splitting scheme similar to Forward1.
- the dimensions of the convolution kernel and the input feature map can be simplified into three dimensions of C (channel), H (height), and W (width).
- Ux Uy ⁇ Uc>1
- Uc M/4 n ,
- the Forward1 convolution splitting scheme can be implemented by the block circuit integrated in the main processing circuit, or the block completely or partially independent of the main processing circuit
- the circuit splits the input feature map and convolution kernel into multiple corresponding split units.
- the block circuit can also convert the dimension storage order of the input feature map and the convolution kernel, so that the data in each split unit is continuously stored as a data row.
- the split and transformed input feature maps and/or convolution kernels may be provided to a master processing circuit or a slave processing circuit.
- the main processing circuit can distribute the data obtained by it to multiple slave processing circuits for performing convolution operations; and perform splicing processing on the operation results returned by the scheduled multiple slave processing circuits according to the convolution splitting scheme, To obtain the output feature map of the convolution operation of the input feature map and the convolution kernel.
- Multiple slave processing circuits can perform convolution operations based on the data they obtain, and return the operation results to the main processing circuit.
- the operation distribution on different C can be carried out relatively independently on different operation circuits.
- the C dimension will be aligned by 4B. Therefore, when processing in units of splitting units, the C dimension will be aligned to 4B (that is, Uc) before splitting. In other words, the processing on different computing circuits is split in units of Uc in the C dimension.
- the number of channels C is usually small, while the convolution kernel and input feature map are generally large.
- the channel C dimension Nc of the input feature map and the convolution kernel in a single round of operation usually has a multiple of Uc that does not exceed the number of scheduled slave processing circuits, so the operation of a single channel calculated in units of Uc can be It is done by one or more slave processing circuits. More generally, even if the C dimension is large, it can be realized by splitting into multiple rounds of operations, wherein the multiple of the C dimension size Nc to Uc of each round of operation processing does not exceed the number of scheduled slave processing circuits.
- the number of calculation rounds required to complete the convolution operation and the number of C processed in each round of operation can be determined Nc or the corresponding grouping mode, where Nc is aligned to Uc.
- the Forward1 solution can also support the three grouping modes described above in conjunction with FIG. 7: Group1, Group4, and Group16.
- the difference between the grouping modes of Forward1 and Forward4 is that the division of the C dimension in Forward1 is performed in units of Uc, for example, every 4 consecutive Cs (corresponding to one Uc) are assigned to a group (or from the processing circuit group SLB).
- Nc is aligned according to Uc, and it is determined that each Rs slave processing circuit processes the same Uc
- the convolution kernel and the input feature map, Rs [Ns/(Nc/Uc)], represent the number of weight multiplexing between the slave processing circuits.
- the convolution kernel can be determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so that in During the operation, it is transmitted to the scheduled multiple slave processing circuits through the broadcast bus.
- the input feature map may be determined as the distribution data, and the distribution data after splitting and transforming the dimension storage sequence is stored in the second storage circuit, so as to be distributed to the corresponding slave processing circuit. These distributed data can be distributed to corresponding slave processing circuits before operation.
- the input feature map can be split among multiple SLs of a single SLB with reference to the schematic diagram of FIG. 8 .
- the input feature map corresponding to Uc can be divided between each Rs slave processing circuits as follows: According to the size of the output feature map, it is divided into Rs output feature blocks with the same shape on the XY dimension; according to the calculation of each The input feature map area required for the output feature block, the input feature map is divided into Rs input feature blocks in the XY dimension; and the Rs input feature blocks are split according to the split unit and the dimension storage order is converted and stored in The second storage circuit is in a storage area allocated by Rs slave processing circuits.
- FIG. 9a For the storage content in the second storage circuit, for example, refer to FIG. 9a (Group1) and FIG. 9c (Group4), and the storage content in the Group16 mode is not shown.
- the first buffer circuit can buffer a plurality of input characteristic data rows from the second storage circuit distributed to the slave processing circuit; and the second buffer circuit can buffer the multicast transmission from the first storage circuit to the slave processing circuit.
- a plurality of weight data rows of the convolution kernel corresponding to the output channel value of the processing circuit.
- these data rows can be distributed to the corresponding computing circuit CU or broadcast to all CUs in the slave processing circuit during the computing period.
- each operation circuit CU can perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the weight data row selected from the second buffer circuit in each operation.
- each operation circuit calculates an output point on the output feature map at the X and/or Y dimension interval; and in different calculations, each operation circuit calculates Different output points in the X and/or Y dimensions on the output feature map.
- FIG. 20 shows a schematic diagram of division of output points of the operation circuit in the Forward1 scheme according to an embodiment of the present disclosure.
- the convolution kernel is split in units of 4 ⁇ 4, only one line of weights in the second buffer circuit needs to be used for each calculation, and the input feature data that can be stored in the first buffer circuit is at most 9 lines, so it can be used at most Computes an 8 ⁇ 8 output.
- the figure shows the division of 4 computing circuits on 8 ⁇ 8 output points, where the output points assigned to the 4 different computing circuits CU0 - CU3 are shown with different backgrounds. Since there is only one row of weights for each calculation, the current output point can be obtained for each calculation without sliding accumulation.
- the 4 CUs when sliding for the first time, respectively calculate the 4 output points in the first sub-block 2001; when sliding to the right for the second time, the 4 CUs respectively calculate the 4 output points in the second sub-block 2002, and so on. 8 ⁇ 8 output points need to slide 16 times accordingly.
- FIG. 21 shows a schematic diagram of a single operation process in the Forward1 scheme according to an embodiment of the present disclosure.
- the size of the first buffer circuit 2110 is 3 ⁇ 3 ⁇ 64B, that is, a maximum of 9 rows of data can be buffered
- the size of the second buffer circuit 2120 is 2 ⁇ 2 ⁇ 64B, that is, a maximum of 4 rows of data can be buffered .
- the storage in the buffer circuit in the figure is also shown in the split unit.
- the figure shows the operation process of the first sliding fetch.
- using the split unit as a sliding window slidingly select N CU input feature lines from the first buffer circuit, and send them to N CU arithmetic circuits in the slave processing circuit for calculation ; Read one weight data row from the second buffer circuit, and broadcast it to N CU arithmetic circuits in the slave processing circuit.
- one input characteristic data row is selected from the first buffer circuit 2110 at the initial position and the position moved by 1 in the X and/or Y directions, and a total of four input characteristic data rows are selected, and correspondingly sent to the slave Four arithmetic circuits 2140 in the processing circuit SL.
- Different backgrounds in 2150 Colored results represent those obtained by different arithmetic circuits 2140 . It can be seen that for each operation, one CU calculates one output point on each XoYo surface on Uc, and the four CUs obtain a total of Uc ⁇ 2 ⁇ 2 output points. It can be seen that the output points calculated by the 4 CUs are adjacent in the XoYo dimension of the output feature map.
- the first buffer circuit slides to fetch the number, and the second buffer circuit does not need to slide, and still uses this row weight for the next calculation.
- the operation circuit splices the Nk*Uc output points calculated during the Nk sliding calculations according to the division method of the output points, and obtains Nk*N CU operation results on the Uc channels.
- the maximum convolution kernel size supported by a single operation of the slave processing circuit is 4 ⁇ 4.
- Fig. 22 shows a schematic diagram of a sliding convolution process in the Forward1 scheme according to an embodiment of the present disclosure.
- This example takes an input feature map of 11 ⁇ 11 and a convolution kernel of 4 ⁇ 4 as an example.
- the convolution step size is 1, and the output feature map size is 8 ⁇ 8.
- the input feature map needs to be aligned to 12 ⁇ 12, divided into 9 blocks of 4 ⁇ 4 ⁇ 4 (C ⁇ H ⁇ W) size, and stored in the first buffer circuit, shown as 2210 in the figure, where the C dimension is omitted .
- the convolution kernel is split according to 4 ⁇ 4 and stored in the second buffer circuit, shown as 2220 in the figure, and the C dimension is also omitted. For each calculation, a 4 ⁇ 4 convolution kernel is selected, which just corresponds to the 4 ⁇ 4 block of the input feature map, and broadcast to 4 computing circuits.
- FIG. 22 The input feature map and the selection range of the convolution kernel in the first buffer circuit and the second buffer circuit for each slide are shown in Figure 22.
- block 2210 represents the input feature map in the first buffer circuit, and the four dotted-line boxes represent the areas selected to be sent to four CUs;
- block 2220 represents the convolution kernel in the second buffer circuit, and the dotted-line frame represents the selected one Weight row, which is broadcast to 4 CUs and does not need to be reselected during sliding.
- the maximum convolution kernel size supported by a single operation of the processing circuit is at least determined by the space sizes of the first buffer circuit and the second buffer circuit, for example. It can be understood that when the convolution kernel exceeds the maximum convolution kernel size, it needs to be split in the Kx and Ky directions according to the maximum convolution kernel size.
- each CU performs bitwise multiplication and accumulation according to 1/Uc row for one input characteristic data row from the first buffer circuit and one weight data row from the second buffer circuit, to obtain each XoYo on Uc 1 output point on the surface, so that N CU computing circuits can get N CU output points on the Uc XoYo surface each time. It can be understood that after sliding through the operation of Nk operation cycles, Nk*N CU output points on Uc XoYo surfaces can be obtained, and they can be spliced to obtain the largest 8 ⁇ 8( Ho*Wo) output points, that is, Uc ⁇ 8 ⁇ 8.
- each CU calculates one output point on the Uc surface in the C dimension once, and the partial sum is 1/Uc(1/4 )
- the result of the bitwise multiplication and accumulation of data rows, that is, each output point is a 4 ⁇ 4 (Y ⁇ X) 2D convolution.
- a single calculation only supports the case of a convolution kernel of 4 ⁇ 4.
- each slave processing circuit SL can convert the operation result of its internal operation circuit CU into a specified format, for example, the format of Nc ⁇ Uy ⁇ Ux.
- each slave processing circuit may output partial operation results of its internal partial operation circuit each time, and these partial operation results are continuous on the X and/or Y dimensions of the output feature map.
- the block circuit may further store the operation results returned from each slave processing circuit in a fourth dimension storage order. According to the situation, the block circuit can also convert the operation result into the desired dimension storage order for storage.
- the output data format is slightly different.
- FIG. 23 shows a schematic diagram of an output data format of the Forward1 scheme according to an embodiment of the present disclosure.
- the grouping mode is Group1
- Figure 2310 shows the raw output of 1 SL. It can be seen from the figure that each SL outputs the area of Uc ⁇ 1 ⁇ 8 (C ⁇ Y ⁇ X) each time, that is, it outputs part of the calculation results of its internal partial operation circuit each time, for example, each of the 2 CUs has 4 operation results (see FIG. 20 ), this part of the operation results is continuous on the X and/or Y dimensions of the output feature map, for example, the same row (as shown in FIG. 20 ) or the same column.
- the area of Uc ⁇ 8 ⁇ 8 (C ⁇ Y ⁇ X) is returned 8 times in a row, that is, 16 operation results of each of the 4 CUs.
- Different SLs output different regions of the output feature map of the same Uc. After outputting all 8 ⁇ 8 regions of Uc, continuing to output will switch different output points.
- the final output data becomes the format of Yo*Xo*ceil[C/Uc]*Uc*8*16*8 after being written into the storage circuit (for example, the first storage circuit), where Yo and Xo are The number of blocks of the output feature map that each SL is divided into, 16 is the division on 16 SLs.
- pendulum operations can be performed again to convert to other desired data formats.
- the output data format is also slightly different. Assuming the original output size is:
- (8*16*8) is the basic output block of forward1, and the directions correspond to h*c*w respectively, where 16 represents the division of ho and wo of the same Uc on 16 SLs; ho and wo are divided twice 4, where the first 4 indicates 4 ⁇ 4 splitting when storing data in SL, and the second 4 indicates data block folding in h and w directions.
- This shape is also the shape of the schematic diagram in FIG. 23 .
- the Group4 output data shape is:
- the Group16 output data shape is:
- the hardware when outputting, can automatically output neurons according to the dimension of 8*16*8 (Y*SL*X) in the row and the dimension of Y*X*C between the rows. The same is true for larger convolution kernels.
- a single slave processing circuit including four operation circuits can calculate up to 16 4 ⁇ 4 output feature regions, so the input feature map/neuron can be multiplexed , thereby reducing the reading frequency of the second storage circuit. That is, the reading frequency of the first storage circuit and the second storage circuit may be different. If the result calculated by the arithmetic circuit is a partial sum, it is stored in the internal register.
- the slave processing circuit can be further used to: determine the input feature multiplexing times rn in the slave processing circuit according to the storage space limitation in the arithmetic circuit; and control the loading frequency of the weight data in the second buffer circuit , so that the input feature data loaded each time in the first buffer circuit is reused rn times, and the convolution operation is performed with the corresponding weight data loaded in the second buffer circuit rn times.
- rn may take a value no greater than 16.
- the shape of the split unit is the same as Forward1, which is also 4B ⁇ 4 ⁇ 4; the difference is that Update1 is applied to the deep convolution operation in the reverse training of the neural network model, specifically for the reverse of the deep convolution operation
- the weight update process in training, and Forward1 is applied to the forward depth convolution operation, both of which are 2D convolution operations.
- the sizes of top_diff and bottom_data are usually relatively large, so different optimization operation schemes are required.
- top_diff and bottom_data will be used to refer to the data to be operated
- the previous description of the convolution kernel can be similarly applied to top_diff
- the description of the input feature map can be similarly applied to bottom_data, also That is, the two can be used interchangeably.
- the description below can be applied in a convolutional splitting scheme similar to Update1.
- top_diff and bottom_data can be simplified into three dimensions of C (channel), H (height), and W (width).
- Ux Uy ⁇ Uc>1
- Uc M/4 n ,
- the multiplication results of the C channel dimension are not accumulated. Therefore, when the operator performs conventional 3D convolution, for example, 64 numbers on the C dimension are multiplied by 64 numbers, and 1 number is obtained after accumulation. Now But it will get 64 numbers. That is to say, the computing power of the calculator is wasted due to non-accumulation in the C dimension, which brings performance loss to the calculator.
- the data on the dimension that needs to be accumulated (such as the HW dimension) is transferred to the C dimension through the above-mentioned splitting method, so that the utilization rate of the calculator can be improved. For example, when the split unit of 4B ⁇ 4 ⁇ 4 is adopted, assuming that the data type is int8, the accumulated result of multiplying 64 numbers by 64 numbers will obtain 4 numbers instead of the original 64 numbers.
- the block circuit integrated in the main processing circuit, or the block completely or partially independent of the main processing circuit The circuit splits bottom_data and top_diff into multiple split units.
- the blocking circuit can also convert the dimension storage order of bottom_data and top_diff, so that the data in each split unit is continuously stored as a data row.
- the split and converted bottom_data and/or top_diff may be provided to a master processing circuit or a slave processing circuit.
- the main processing circuit can distribute the data obtained by it to multiple slave processing circuits for performing convolution operations; and perform splicing processing on the operation results returned by the scheduled multiple slave processing circuits according to the convolution splitting scheme, To obtain the output ⁇ W (or weight_diff) of the depth convolution operation of bottom_data and top_diff.
- Multiple slave processing circuits can perform convolution operations based on the data they obtain, and return the operation results to the main processing circuit.
- the C dimension is usually larger, such as greater than 64, and bottom_data and top_diff are generally larger.
- the size Nc of the channel C dimension of bottom_data and top_diff in a single-round operation can be a multiple of 64, so the operation of a single channel calculated in units of Uc can be allocated to one slave processing circuit to complete. Therefore, in some embodiments, the convolution splitting scheme also indicates the group division method for performing the deep convolution operation, wherein the group division method can be sequentially divided into There are Ns schedulable slave processing circuits, each of which processes bottom_data and top_diff data of different continuous Uc C values. In other words, a single slave processing circuit can be a group, respectively processing operations of different C (in Uc units), that is, corresponding to the aforementioned Group16 grouping mode.
- the top_diff may be split according to the aforementioned convolution splitting scheme, and the dimensions may be converted and then stored in the first storage circuit. Since each slave processing circuit processes a different Uc, top_diff corresponding to different Uc C values can be unicast/respectively transmitted to the scheduled Ns slave processing circuits through the broadcast bus during operation.
- the bottom_data can be determined as the distribution data, and the distribution data after splitting and transforming the dimension storage order are respectively stored in the second storage circuit in a manner of sequentially dividing the channel C dimension and taking Uc as the unit.
- the storage area corresponding to the Ns slave processing circuits so as to distribute to the corresponding slave processing circuits.
- Fig. 24 shows a schematic storage manner of bottom_data in the second storage circuit according to some embodiments of the present disclosure.
- the second storage circuit may allocate a storage area to each slave processing circuit, so that the bottom_data data required for operation of each slave processing circuit only needs to be read from its corresponding storage area.
- the figure exemplarily shows that 16 storage areas 2400 - 2415 are allocated to 16 slave processing circuits, and each storage area stores the data block of bottom_data to be processed by the slave processing circuit.
- the C dimension is split in units of Uc.
- the size of the C dimension exceeds Uc times the number of schedulable slave processing circuits, multiple calculation rounds are required to execute the calculation.
- bottom_data can be divided into 32 bottom_data data blocks according to the C dimension and Uc as the unit. The first 16 data blocks are calculated in the first round of calculation, and the last 16 data blocks are calculated in the second round of calculation.
- the bottom_data data block is similarly divided and stored accordingly, and will not be repeated here.
- the first buffer circuit can buffer a plurality of bottom_data data lines distributed to the slave processing circuit from the second storage circuit; and the second buffer circuit can buffer the unicast transmission from the first storage circuit to the slave processing circuit A plurality of top_diff data rows corresponding to Uc of the circuit.
- these data rows can be distributed to the corresponding computing circuit CU or broadcast to all CUs in the slave processing circuit during the computing period.
- each operation circuit CU can perform a bit-wise multiply-accumulate operation on the bottom_data data row selected from the first buffer circuit and the top_diff data row selected from the second buffer circuit in each operation.
- the output points need to be split among the multiple CUs. Similar to Forward4, in Update1, it is also divided according to the way that each computing circuit allocates interval output points (for example, Figure 10b). In Update1, the convolution kernel top_diff is split in units of 4 ⁇ 4, and the bottom_data data only uses 2 ⁇ 2 64B in the first buffer circuit at a time. Therefore, multiple sliding calculations are performed on the first buffer circuit After that, up to 4 ⁇ 4 output points can be calculated.
- each arithmetic circuit calculates an output point on the XY plane of the Uc channel C values of the output ⁇ W, adjacent in the X and/or Y dimension; and In different calculations, each operation circuit calculates different output points on the output ⁇ W in the X and/or Y dimensions.
- the single operation process in the Update1 scheme can be similar to Forward1, and can refer to the description in conjunction with FIG. 21 , which will not be repeated here.
- Fig. 25 shows a schematic diagram of a sliding convolution process in the Update1 scheme according to an embodiment of the present disclosure.
- Each row of data is a block of size 4x4x4 (CxHxW).
- the top_diff with a size of 4 ⁇ 4 is selected from the second buffer circuit, which just corresponds to the 4 ⁇ 4 block of bottom_data, and broadcast to the 4 calculation circuits.
- FIG. 25 The selection ranges of bottom_data and top_diff in the first buffer circuit and the second buffer circuit during each slide are shown in FIG. 25.
- block 2510 represents the bottom_data in the first buffer circuit, and the four dotted-line boxes represent the areas selected to be sent to four CUs;
- block 2520 represents top_diff in the second buffer circuit, and the dotted-line boxes represent the selected top_diff data row, It is broadcast to 4 CUs, and no reselection is required during sliding.
- Update1 convolution operation mode the maximum ⁇ W size supported by a single operation of the slave processing circuit is 4 ⁇ 4. It can be understood that when ⁇ W exceeds the maximum supported size, it needs to be split in the XY direction according to the maximum supported size.
- each CU will correspond to bottom_data and top_diff of the same channel value in units of 1/Uc data lines for one bottom_data data line from the first buffer circuit and one top_diff data line from the second buffer circuit
- the data is multiplied and accumulated to obtain Uc output points, that is, 1 output point of ⁇ W on each KxKy surface of Uc, so that N CU arithmetic circuits can obtain N CU output points on Uc KxKy surfaces each time . It can be understood that after sliding through Nk operation cycles, each operation circuit calculates and obtains Nk output points spaced apart in the X and/or Y dimensions on the Uc KxKy plane.
- a total of Nk slides of N CU arithmetic circuits can obtain Nk*N CU output points on Uc KxKy surfaces. These output points can be spliced to form a maximum of 4 ⁇ 4 (Kx*Ky) output points on the Uc surface in the C dimension, that is, Uc ⁇ 4 ⁇ 4.
- each CU calculates one output point on the Uc surface in the C dimension once, and the partial sum is 1/Uc(1/4 )
- the result of the bitwise multiplication and accumulation of data rows, that is, each output point is a 4 ⁇ 4 (Y ⁇ X) 2D convolution.
- the calculation of the maximum output point is completed, and an output of 4 ⁇ 4 (Y ⁇ X) is obtained in one SL (as shown in FIG. 10 b ).
- each slave processing circuit SL can convert the operation result of its internal operation circuit CU into a specified format.
- each slave processing circuit can output one output point at the same position on the Uc XY plane obtained by one operation circuit within it each time.
- the Ns slave processing circuits simultaneously output one output point at the same position on the Ns*Uc XY plane each time. Through this output mode, the Ns*Uc output points are continuous in the C dimension.
- the block circuit may further store the operation results returned from each slave processing circuit in a fourth-dimensional storage order, for example, concatenate and store in the order of Ky*Kx*(Ns*Uc) dimensions. According to the situation, the block circuit can also convert the operation result into the desired dimension storage order for storage.
- FIG. 26 shows a schematic diagram of an output data format of the Update1 scheme according to an embodiment of the present disclosure.
- a method of dividing groups according to the C dimension is adopted, that is, each slave processing circuit SL processes operations of different Uc.
- each SL outputs the area of Uc ⁇ 1 ⁇ 1 (C ⁇ Y ⁇ X) each time, that is, it outputs the Uc operation results of one operation circuit in it each time, for example, the 4 operations of CU0 As a result, these 4 operation results are continuous in the C dimension of the output data. Since different SLs handle different Uc operations, 16 SLs can output 1 output point at the same position on different Ucs and XY planes at the same time, which can be spliced into 16*Uc output points in the C dimension, which in the C dimension continuous.
- 2620 in the figure shows the deposit-out data structure of 16 SLs.
- the final output data becomes in the format of Kh*Kw*(16*Uc), where 16 is the division on 16 SLs.
- pendulum operations can be performed again to convert to other desired data formats.
- a single slave processing circuit including four operation circuits can calculate up to 16 4 ⁇ 4 output point areas, so the bottom_data can be multiplexed, thereby reducing the second The reading frequency of the memory circuit. That is, the reading frequency of the first storage circuit and the second storage circuit may be different. If the result calculated by the arithmetic circuit is a partial sum, it is stored in an internal register.
- the slave processing circuit can be further used to: determine the bottom_data multiplexing times rn in the slave processing circuit according to the storage space limitation in the arithmetic circuit; and control the loading frequency of the top_diff data in the second buffer circuit to The bottom_data data loaded each time in the first buffer circuit is reused rn times, and the convolution operation is performed with the corresponding tap_diff data loaded rn times in the second buffer circuit.
- rn may take a value no greater than 16.
- Update4 the shape of the split unit is the same as that of Update1, which is also 4B ⁇ 4 ⁇ 4; the difference is that Update4 is applied to the cross product convolution operation in the reverse training of the neural network model, specifically for the cross product convolution operation
- Update4 is applied to the cross product convolution operation in the reverse training of the neural network model, specifically for the cross product convolution operation
- the weight update process in the reverse training, and Update1 is applied to the reverse depth convolution operation.
- the principle of the reverse cross-product convolution operation reference may be made to the previous description in conjunction with FIG. 4c. Due to the characteristics of the reverse cross product convolution operation, different optimization operation schemes are required.
- top_diff and bottom_data will be used to refer to the data to be operated
- the previous description of the convolution kernel can be similarly applied to top_diff
- the description of the input feature map can be similarly applied to bottom_data, also That is, the two can be used interchangeably.
- the description below can be applied in a convolution splitting scheme similar to Update4.
- Ux Uy ⁇ Uc>1
- the block circuit integrated in the main processing circuit or the block completely or partially independent of the main processing circuit can be used according to the Update4 convolution split scheme.
- the circuit splits bottom_data and top_diff into multiple corresponding split units.
- the blocking circuit can also convert the dimension storage order of bottom_data and top_diff, so that the data in each split unit is continuously stored as a data row.
- the split and converted bottom_data and/or top_diff may be provided to a master processing circuit or a slave processing circuit.
- the main processing circuit can distribute the data obtained by it to multiple slave processing circuits for performing convolution operations; and perform splicing processing on the operation results returned by the scheduled multiple slave processing circuits according to the convolution splitting scheme, To obtain the output ⁇ W (or weight_diff) of the depth convolution operation of bottom_data and top_diff.
- Multiple slave processing circuits can perform convolution operations based on the data they obtain, and return the operation results to the main processing circuit.
- the output ⁇ W (weight gradient) includes four dimensions [Co Kh Kw Ci], where the results of operations on the Co dimension are relatively independent . Therefore, the operation distribution of different Co can be performed relatively independently on different operation circuits.
- the C dimension will be aligned according to 4B. Therefore, when processing in units of splitting units, the C dimension will be aligned to 4B (that is, Uc) before splitting. In other words, the processing on different computing circuits is split in units of Uc in the Co dimension.
- the calculation rounds for completing the cross product convolution operation and the output channel Co processed in each round of operation can be determined.
- different grouping modes may be used to perform the convolution operation.
- Co is small, for example, 1-4
- the Group1 grouping mode can be adopted, that is, all slave processing circuits SL belong to one group and jointly process operations of the same Co (that is, one Uc).
- Co is relatively large, such as 4 to 16
- the Group4 grouping mode can be used, that is, all SLs are divided into 4 groups, and each group handles the operation of one Co (that is, one Uc).
- a Group16 grouping mode may be used, that is, each SL is a group, and each SL handles operations of different Cos (that is, different Ucs).
- the grouping modes suitable for different Co ranges are exemplarily described above, they may not be selected according to the above rules.
- the Group1 grouping mode can also be used to complete the required processing through multiple rounds of operations. It can be seen that the splitting mode among different groups (for example, Group1, Group4, Group16) is determined according to Co.
- each group may also be assigned computing tasks, for example, according to the Ci dimension.
- the bottom_data data can be sequentially divided into Rs slave processing circuits in the same group in units of Uc according to the direction of the input channel Ci.
- the top_diff data requires no additional processing.
- the top_diff data can be determined as multicast data, and the multicast data after splitting and converting the dimension storage order can be stored in the first storage circuit, so that during the operation, the data corresponding to different Uc Co values can be transmitted through the broadcast bus.
- the top_diff data is transmitted to the scheduled N slave processing circuit groups respectively, and each slave processing circuit group shares the same neuron gradient data of Uc and Co values.
- the bottom_data data can be determined as the distribution data, and N copies of the distribution data after splitting and transforming the dimension storage sequence are copied, and each copy is divided into Rs data blocks according to the group division method in the Ci direction, and stored in the second In the corresponding storage area in the storage circuit, so as to distribute to the corresponding slave processing circuit.
- top_diff can be directly split by split unit and stored in the first storage circuit; bottom_data can be split by split unit In addition, it is also divided into Ns data blocks in units of Uc according to the Ci dimension and stored in the second storage circuit for distribution to Ns slave processing circuits.
- the master processing circuit may determine bottom_data as the distribution data, and store the distribution data after splitting and converting the dimension storage order in the second storage circuit, so as to distribute to corresponding slave processing circuits.
- Fig. 27a shows exemplary storage contents in the second storage circuit in the Group1 mode in the Update4 scheme according to some embodiments of the present disclosure.
- bottom_data is stored in the second storage circuit, which includes 16 storage areas 2700-2715, which are allocated to 16 slave processing circuits SL0-SL15 respectively.
- top_diff can also be directly divided by the division unit and stored in the first storage circuit. Since each SLB processes a different Co, the top_diff of different Cos can be unicast to the corresponding SLB, and the SLs in the SLB share the same Co. In other words, the top_diff of the same Co will be multicast to multiple SLs in one SLB.
- Fig. 27b shows exemplary storage contents in the second storage circuit when divided according to the C dimension in the Group4 mode in the Update4 scheme according to some embodiments of the present disclosure.
- the second storage circuit also includes 16 storage areas 2700-2715, which are allocated to 16 slave processing circuits SL0-SL15 respectively.
- the 16 storage areas are also divided into 4 groups according to the corresponding SLBs, and each group stores the same and complete bottom_data, that is, 4 copies of bottom_data are stored in the storage areas corresponding to the 4 SLBs.
- each SLB processes the top_diff of the same bottom_data and different Cos; and the four SLs in each SLB respectively process a split bottom_data data block.
- These bottom_data data blocks are split according to the Ci dimension, specifically, according to the Ci dimension interval of 1 Uc, they are sequentially allocated to the corresponding storage areas of the 4 SLs in one SLB. Therefore, the storage contents of the storage areas for the four SLBs in the figure are the same, for example, the contents in 2700-2703 are the same as the contents in 2712-2715.
- the same storage allocation is also performed in the storage areas of other SLBs, which will not be repeated here.
- the 16 SLs can be divided according to the Ci dimension. For this part, you can refer to the description of the previous Update1 solution, and will not repeat it here.
- the first buffer circuit can cache a plurality of bottom_data data lines distributed to the slave processing circuit from the second storage circuit; and the second buffer circuit can cache the unicast, multicast or broadcast data from the first storage circuit A plurality of top_diff data lines corresponding to Co (in Uc) transmitted to the slave processing circuit.
- these data rows can be distributed to the corresponding computing circuit CU or broadcast to all CUs in the slave processing circuit during the computing period.
- each operation circuit CU can perform a bit-wise multiply-accumulate operation on the bottom_data data row selected from the first buffer circuit and the top_diff data row selected from the second buffer circuit in each operation.
- each computing circuit calculates and outputs ⁇ W differently on Co, on one Ci with Uc as the unit (that is, on consecutive Uc Ci values), and on the same XY dimension 1 output point for the location.
- each calculation circuit calculates different output points on the output ⁇ W in the XY dimension.
- the number of slides Nk Kx*Ky, where Kx and Ky are the size of the output ⁇ W in the X and Y dimensions or the smaller value of the maximum output size supported by a single operation of the processing circuit in the current convolution split mode .
- FIG. 28 shows a schematic diagram of a single operation process in the Update4 solution according to an embodiment of the present disclosure.
- the size of the first buffer circuit 2810 is 3 ⁇ 3 ⁇ 64B, that is, a maximum of 9 rows of data can be buffered
- the size of the second buffer circuit 2820 is 2 ⁇ 2 ⁇ 64B, that is, a maximum of 4 rows of data can be buffered .
- the storage in the buffer circuit in the figure is also shown in the split unit.
- the figure shows the operation process of the first sliding fetch.
- take the split unit as the sliding window, slide and select 1 bottom_data data row from the first buffer circuit, and broadcast and transmit it to NCU arithmetic circuits in the slave processing circuit for calculation;
- Read 1 top_diff data line from the second buffer circuit split Co (one Uc) into Uc Cos, copy Uc parts on the XY data surface of each Co, and send them to the Uc arithmetic circuits in the slave processing circuit respectively .
- N CU 4
- a bottom_data data line is selected at the starting position from the first buffer circuit 2810 and broadcast to the four arithmetic circuits 2840 in the slave processing circuit SL.
- Select one top_diff data row at the starting position from the second buffer circuit 2820 that is, select the data 2830 of 4 ⁇ 4 ⁇ 4 size, and split it into four 1 ⁇ 4 ⁇ 4 data planes in the Co dimension , each data plane is copied in 4 copies, expanded into 4 ⁇ 4 ⁇ 4 (Ho ⁇ Wo ⁇ Ci) data lines, and sent to the 4 arithmetic circuits 2840 in the SL respectively.
- the bottom_data data and top_diff data of the same input channel Ci value are multiplied and accumulated to obtain Uc output points of the assigned Co value in the Ci dimension.
- four computing circuits 2840 perform bitwise multiplication and accumulation operations on the broadcast bottom_data data row and the distributed top_diff extended data row according to 1/4 row, and obtain the operation result 2850.
- the results of different background colors in 2850 represent different Arithmetic circuit 2840 obtained. It can be seen that for each operation, one CU calculates one output point on each KxKy plane on the Uc (Ci dimension) of a Co allocated, and four CUs obtain four 1 ⁇ 1 ⁇ Uc output points on Co in total. . It can be seen that the output points calculated by the four CUs correspond to the same position on the KxKy dimension of different Cos.
- each operation circuit calculates Nk*Uc output points during Nk sliding calculations, which are Nk output points continuous in X and/or Y dimensions on a single Co and Uc Ci on the XY plane .
- the 4 computing circuits can obtain Nk computing results on the XY plane on the 4 Co, Uc, and Ci in total.
- the maximum output size supported by a single operation of the slave processing circuit is 4 ⁇ 4.
- FIG. 29 shows a schematic diagram of a sliding convolution process in the Update4 scheme according to an embodiment of the present disclosure.
- the first buffer circuit buffers 2*2 4 bottom_data data rows, shown as 2910 in the figure, where the C dimension is omitted; the second buffer circuit buffers 1 top_diff data row, shown in the figure For 2920, the C dimension is also omitted.
- Each row of data is a block of size 4x4x4 (CxHxW).
- the top_diff with a size of 4 ⁇ 4 is selected from the second buffer circuit, split and copied according to C, expanded into 4 data rows, and distributed to 4 operation circuits.
- FIG. 29 The selection ranges of bottom_data and top_diff in the first buffer circuit and the second buffer circuit during each slide are shown in FIG. 29 , and there are 16 pictures in total, representing a total of 16 slides.
- block 2910 represents bottom_data in the first buffer circuit, and the dotted line box represents the area selected for broadcasting to four CUs;
- block 2920 represents top_diff in the second buffer circuit, and the dotted line box represents a selected top_diff data line, which is selected After copying and expanding, it is distributed to 4 CUs, and there is no need to reselect during the sliding process.
- the maximum ⁇ W size supported by a single operation of the slave processing circuit is 4 ⁇ 4. It can be understood that when ⁇ W exceeds the maximum supported size, it needs to be split in the XY direction according to the maximum supported size.
- each CU performs bitwise multiplication and accumulation according to 1/Uc row for a bottom_data data row from the first buffer circuit and a top_diff extended data row from the second buffer circuit, and obtains ⁇ W in a Co One output point on each KxKy plane on Uc and Ci, so that N CU arithmetic circuits can obtain one output point on N CU Co and Uc KxKy planes each time.
- Kx*Ky output points on N CU Co and Uc KxKy surfaces can be obtained, and they can be spliced to obtain N CU Co and Uc A maximum of 4 ⁇ 4 (Kx*Ky) output points on the surface, that is, Ky ⁇ Kx ⁇ N CU ⁇ Uc(Ky ⁇ Kx ⁇ Co ⁇ Ci).
- each CU calculates one output point on Co and Uc surfaces in the Ci dimension at a time, and the output point is 1/Uc (1/4)
- each slave processing circuit SL can convert the operation result of its internal operation circuit CU into a specified format.
- each slave processing circuit can output one operation result of one operation circuit in it each time, and these operation results are one output of the output data at the same position on the XY plane on one Co and Uc Ci. point. That is, these Uc output points are continuous in the Ci dimension.
- the Rs slave processing circuits in the same SLB simultaneously output one output point at the same position on the XY plane of the Rs*Uc Cis of the same Co each time.
- the block circuit can further store the operation results returned from each slave processing circuit in the fourth dimension storage order, for example, splicing and storing according to the Ky*Kx*Co/N*N*(Rs*Uc) dimension order, where N represents GroupN , that is, the number of groups. According to the situation, the block circuit can also convert the operation result into the desired dimension storage order for storage.
- the output data format is slightly different.
- FIG. 30 shows a schematic diagram of an output data format of the Update4 solution according to an embodiment of the present disclosure.
- the Group1 grouping mode is adopted, and the grouping is divided according to the Ci dimension, that is, each slave processing circuit SL processes the same Co (in Uc as the unit) and different Ci (in Uc as the unit) output data operation .
- each SL outputs an area of 1 ⁇ Uc ⁇ 1 ⁇ 1 (Co ⁇ Ci ⁇ Y ⁇ X) each time, that is, it outputs the operation result of one operation circuit in it each time, such as the calculation result of CU0
- FIG. 3020 in the figure shows the deposit-out data structure of 16 SLs.
- the outputs of 16 SLs are concatenated into a continuous row of data in the Ci dimension each time.
- the final output data becomes in the format of Kh*Kw*Co*(16*Uc), where 16 is the division on 16 SLs.
- pendulum operations can be performed again to convert to other desired data formats.
- the output data format is slightly different, which can be expressed as Ky*Kx*Co/N*N*(Rs*Uc), where N is GroupN The number of groups in .
- Update4 uses 4B*4*4 blocks as calculation units, it is inevitable that there will be alignment restrictions during calculation. According to different grouping modes (such as Group1, Group4, Group16, etc.), the final alignment restrictions during calculation are also different. Those skilled in the art can calculate the alignment constraints for each data according to different data types and different grouping modes, which will not be described in detail here.
- a single slave processing circuit including four operation circuits can calculate up to 16 4 ⁇ 4 output point areas, so the bottom_data can be multiplexed, thereby reducing the second The reading frequency of the memory circuit. That is, the reading frequency of the first storage circuit and the second storage circuit may be different. If the result calculated by the arithmetic circuit is a partial sum, it is stored in an internal register.
- the slave processing circuit can be further used to: determine the bottom_data multiplexing times rn in the slave processing circuit according to the storage space limitation in the arithmetic circuit; and control the loading frequency of the top_diff data in the second buffer circuit to The bottom_data data loaded each time in the first buffer circuit is reused rn times, and the convolution operation is performed with the corresponding top_diff data loaded in the second buffer circuit rn times.
- rn may take a value no greater than 16.
- Embodiments of the present disclosure also provide a method for performing a convolution operation by using the aforementioned computing device.
- steps of the method for performing the convolution operation correspond to the various circuits of the computing device described above in conjunction with the accompanying drawings, so the features described above are also applicable to the steps of the method and will not be repeated here.
- An embodiment of the present disclosure also provides a chip, which may include the computing device in any embodiment described above with reference to the accompanying drawings. Further, the present disclosure also provides a board, which may include the aforementioned chip.
- the electronic equipment or devices disclosed in this disclosure may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC equipment, IoT terminals, mobile Terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, earphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, vehicles, household appliances, and/or medical equipment.
- Said vehicles include airplanes, ships and/or vehicles;
- said household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, range hoods;
- said medical equipment includes nuclear magnetic resonance instruments, Ultrasound and/or electrocardiograph.
- the electronic equipment or device disclosed herein can also be applied to fields such as the Internet, the Internet of Things, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical care. Further, the electronic device or device disclosed herein can also be used in application scenarios related to artificial intelligence, big data and/or cloud computing, such as cloud, edge, and terminal.
- electronic devices or devices with high computing power according to the disclosed solutions can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and/or Edge devices (such as smartphones or cameras).
- the hardware information of the cloud device and the hardware information of the terminal device and/or the edge device are compatible with each other, so that according to the hardware information of the terminal device and/or the edge device, the hardware resources of the cloud device can be Match appropriate hardware resources to simulate the hardware resources of terminal devices and/or edge devices, so as to complete the unified management, scheduling and collaborative work of device-cloud integration or cloud-edge-end integration.
- the present disclosure expresses some methods and their embodiments as a series of actions and combinations thereof, but those skilled in the art can understand that the solution of the present disclosure is not limited by the order of the described actions . Therefore, according to the disclosure or teaching of the present disclosure, those skilled in the art may understand that certain steps may be performed in other orders or simultaneously. Further, those skilled in the art can understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily required for the realization of one or some solutions of the present disclosure. In addition, according to different schemes, the description of some embodiments in this disclosure also has different emphases. In view of this, those skilled in the art may understand the part that is not described in detail in a certain embodiment of the present disclosure, and may also refer to related descriptions of other embodiments.
- a unit described as a separate component may or may not be physically separated, and a component shown as a unit may or may not be a physical unit.
- the aforementioned components or units may be located at the same location or distributed over multiple network units.
- some or all of the units may be selected to achieve the purpose of the solutions described in the embodiments of the present disclosure.
- multiple units in the embodiments of the present disclosure may be integrated into one unit, or each unit exists physically independently.
- the above-mentioned integrated units may also be implemented in the form of hardware, that is, specific hardware circuits, which may include digital circuits and/or analog circuits.
- the physical realization of the hardware structure of the circuit may include but not limited to physical devices, and the physical devices may include but not limited to devices such as transistors or memristors.
- various devices such as computing devices or other processing devices described herein may be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs.
- the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistance memory (Resistive Random Access Memory, RRAM), dynamic Random Access Memory (Dynamic Random Access Memory, DRAM), Static Random Access Memory (Static Random Access Memory, SRAM), Enhanced Dynamic Random Access Memory (Enhanced Dynamic Random Access Memory, EDRAM), High Bandwidth Memory (High Bandwidth Memory , HBM), hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.
- RRAM variable resistance memory
- DRAM Dynamic Random Access Memory
- SRAM Static Random Access Memory
- EDRAM Enhanced Dynamic Random Access Memory
- High Bandwidth Memory High Bandwidth Memory
- HBM High Bandwidth Memory
- HMC Hybrid Memory Cube
- ROM and RAM etc.
- a computing device configured to perform a convolution operation, the computing device comprising:
- a main processing circuit the main processing circuit is used to: obtain an input feature map and/or a convolution kernel, wherein the input feature map and the convolution kernel have been split into multiple split units according to the convolution split scheme and Convert its dimension storage order, wherein the convolution splitting scheme is determined according to the size of the lowest storage dimension before splitting the input feature map, the convolution splitting scheme indicates the shape of the split unit, one The amount of data contained in a split unit does not exceed the maximum single operation of the hardware, and the data in a split unit is continuously stored as a data row; and
- a plurality of slave processing circuits are used to perform convolution operations on the input feature map and the corresponding division units of the convolution kernel.
- Clause A2 The computing device of Clause A1, wherein the convolution splitting scheme is determined as follows:
- Clause A3 The computing device according to Clause A1, further comprising a block circuit for splitting and storing the input feature map and the convolution kernel as follows:
- the split unit From the data to be operated stored in the storage order of the first dimension, take the split unit as a unit, read one or more split units in the first reading order, and store the read split units in the corresponding On the storage circuit, the data in each split unit is stored according to the storage order of the second dimension, and the data between the split units is stored according to the storage order of the third dimension.
- the storage order of the first dimension is HWC from high to low;
- the storage order of the second dimension is CHW from high to low
- the first reading sequence is HWC from high to low
- the storage order of the third dimension is the same as the storage order of the first dimension
- H is the height dimension
- W is the width dimension
- C is the channel dimension
- the number of calculation rounds required to complete the convolution operation, the number of Co processed in each round of operation, or the corresponding grouping mode is determined.
- Clause A6 The computing device according to Clause A5, wherein the grouping mode is GroupN, which means that all slave processing circuits scheduled in the current round of operations are divided into N groups, and each slave processing circuit group processes the same Co value, and different slave processing circuits
- each group of slave processing circuits includes Rs slave processing circuits
- the master processing circuit is further configured to divide among the Rs slave processing circuits as follows Input feature map:
- the output feature map is evenly divided into Rs output feature blocks of the same shape in the HW dimension;
- the input feature map is divided into Rs input feature blocks in the HW dimension, so as to be allocated to the Rs slave processing circuits.
- Clause A8 The computing device of clause A7, wherein the input feature blocks divided are aligned in the YX dimension of the split unit in the HW dimension.
- Clause A9 The computing device of any one of clauses A7-A8, further comprising a first storage circuit and a second storage circuit,
- One of the input feature map and the convolution kernel is determined as multicast data, and the split multicast data is stored in the first storage circuit;
- the other of the input feature map and the convolution kernel is determined as distribution data, and the split distribution data is stored in the second storage circuit.
- the convolution kernel allocated to each slave processing circuit is stored in a corresponding storage area in the second storage circuit.
- each of said slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:
- the first buffer circuit is used for buffering a plurality of input feature rows corresponding to the slave processing circuit from one of the first storage circuit and the second storage circuit;
- the second buffer circuit is used to buffer a plurality of weight rows corresponding to the slave processing circuit from the other of the first storage circuit and the second storage circuit;
- Each operation circuit is configured to perform a bitwise multiply-accumulate operation on the input feature row selected from the first buffer circuit and the weight value row selected from the second buffer circuit during each calculation.
- each said slave processing circuit is further configured to:
- the splitting unit uses the splitting unit as a sliding window, slidingly select N CU input feature lines from the first buffer circuit, and send them to the slave processing circuit respectively
- the N CU arithmetic circuits within are used for calculation;
- Nk is determined according to the size of the convolution kernel in the X and Y dimensions and the maximum convolution kernel size supported by a single operation of the processing circuit in the convolution split mode.
- each said arithmetic circuit is further configured to:
- the Nk*Nop parts calculated in the Nk sliding calculations are accumulated according to the corresponding convolution output points to obtain Nop operation results.
- each of said slave processing circuits is further configured to:
- the output points calculated by the plurality of operation circuits therein are output in a specific order, so that the output points output continuously are continuous in the X and/or Y dimensions.
- Clause A16 The computing device according to any one of clauses A12-A15, wherein the division of output points between the plurality of arithmetic circuits comprises any of the following:
- each arithmetic circuit computes a plurality of output points contiguous in the X and/or Y dimensions;
- Each arithmetic circuit computes a plurality of output points spaced in the X and/or Y dimensions.
- Clause A17 The computing device of Clause A3, wherein the blocking circuit is further configured to:
- Clause A18 The computing device of Clause A3 or A17, wherein:
- the blocking circuit is integrated in the main processing circuit
- the blocking circuit is independent of the main processing circuit.
- the blocking circuit performs the splitting on both the input feature map and the convolution kernel
- the blocking circuit performs the splitting only on data determined to be multicast data in the input feature map and convolution kernel.
- Clause A20 A chip comprising the computing device according to any one of clauses A1-A19.
- Clause A22 A method of performing a convolution operation using the computing device of any one of Clauses A1-A19.
- a processing circuit for performing a convolution operation comprising a first buffer circuit, a second buffer circuit, and a plurality of operation circuits, wherein:
- the first buffer circuit is used to buffer a plurality of input feature lines to be operated
- the second buffer circuit is used for buffering multiple weight rows to be calculated.
- Each of the operation circuits is configured to perform a bitwise multiply-accumulate operation on the input feature row selected from the first buffer circuit and the weight value row selected from the second buffer circuit during each calculation, Wherein the weight row is an extended weight row copied and extended from a local weight row selected from the second buffer circuit.
- Nk is determined according to the smaller value of the size of the convolution kernel in the X and Y dimensions and the maximum convolution kernel size supported by a single operation of the processing circuit in the current convolution operation mode .
- each said arithmetic circuit is further configured to:
- bitwise multiplication and accumulation is performed in units of 1/Nop rows to obtain Nop partial sums
- the Nk*Nop parts calculated in the Nk sliding calculations are accumulated according to the corresponding convolution output points to obtain Nop operation results.
- the output points calculated by the plurality of operation circuits therein are output in a specific order, so that the output points output continuously are continuous in the X and/or Y dimensions.
- Clause B6 The processing circuit according to any one of clauses B2-B5, wherein the division of output points between the plurality of arithmetic circuits includes any of the following:
- each arithmetic circuit computes a plurality of output points contiguous in the X and/or Y dimensions;
- each arithmetic circuit computes a plurality of output points spaced in the X and/or Y dimension.
- each of said input feature row and said weight value row is composed of a split unit, and one said split unit contains the lowest storage dimension and at least Data of an additional storage dimension.
- Clause B10 A computing device configured to perform a convolution operation, said computing device comprising a master processing circuit and a plurality of slave processing circuits, each of said slave processing circuits being configured according to any one of clauses B1-B9. processing circuit.
- Clause B11 A chip comprising the computing device according to Clause B10.
- Clause B13 A method of performing a convolution operation using the processing circuit of any one of clauses B1-B9.
- a computing device configured to perform a convolution operation, the computing device comprising:
- the split and transformed input feature maps and/or convolution kernels are provided to a master processing circuit or a slave processing circuit;
- the main processing circuit is used to distribute the data obtained by it to multiple slave processing circuits for performing convolution operations; Splicing processing to obtain the output feature map of the convolution operation of the input feature map and the convolution kernel; and
- the plurality of slave processing circuits are used to perform convolution operations according to the data obtained by them, and return the operation results to the main processing circuit.
- Clause C2 The computing device of clause C1, wherein the convolution splitting scheme further indicates the number of operation rounds in which the convolution operation is performed, wherein the number of output channels Co processed in each operation round corresponds to the operation The number Ns of slave processing circuits that can be scheduled in a round.
- Clause C3 The computing device of clause C2, wherein said computing device further comprises a first storage circuit and a second storage circuit,
- the input feature map is determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so as to be transmitted to the scheduled multiple slave processing circuits through the broadcast bus during operation;
- the convolution kernel is determined as distribution data, and the distribution data after splitting and converting the dimension storage order is stored in the second storage circuit, so as to be distributed to corresponding slave processing circuits before operation.
- Clause C4 The computing device according to Clause C3, wherein convolution kernels with different Co values assigned to each slave processing circuit in each calculation round are respectively stored in the second storage circuit for the corresponding slave processing circuit. in the storage area.
- each of said slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:
- the first buffer circuit is used for buffering a plurality of input characteristic data rows transmitted by broadcast from the first storage circuit
- the second buffer circuit is used for buffering a plurality of weight data rows from the second storage circuit distributed to the convolution kernel of the slave processing circuit;
- Each operation circuit is used to perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the weight data row selected from the second buffer circuit in each operation.
- each operation circuit calculates a plurality of output points continuous in X and/or Y dimensions on the output feature map.
- Nk Kx*Ky, Kx and Ky are the sizes of the convolution kernel in the X and Y dimensions respectively or the single operation supported by the slave processing circuit in the convolution split mode The smaller value of the maximum kernel size.
- each said arithmetic circuit is further configured to:
- the Nk*Nop parts calculated in the Nk sliding calculations are accumulated according to the corresponding convolution output points to obtain Nop operation results.
- each said slave processing circuit is further configured to:
- each arithmetic circuit computes an output feature block comprising 2x2 output points.
- Clause C15 The computing device according to clause C7, wherein the maximum convolution kernel size supported by a single operation of the slave processing circuit in the convolution split mode is 3 ⁇ 3.
- Clause C16 A chip comprising the computing device according to any one of clauses C1-C15.
- Clause C17 A board comprising the chip according to Clause C16.
- Clause C18 A method of performing a convolution operation using the computing device of any one of Clauses C1-C15.
- a computing device configured to perform a convolution operation, the computing device comprising:
- a main processing circuit the main processing circuit is used to: obtain an input feature map and/or a convolution kernel, wherein the input feature map and the convolution kernel have been split into multiple split units according to the convolution split scheme and Convert the storage order of dimensions, where one split unit includes data of the lowest storage dimension and at least one other storage dimension, and the data volume of a split unit does not exceed the maximum single operation of the hardware, the output channel of the convolution kernel in a single round of operation
- the size of the Co dimension does not exceed the number of said slave processing circuits, and the data in one split unit is continuously stored as one data row; and
- a plurality of slave processing circuits are used to perform convolution operations on the input feature map and corresponding data rows of the convolution kernel.
- Clause D2 The computing device of Clause D1, wherein the convolution splitting scheme further indicates the number of operation rounds in which the convolution operation is performed, the number of Cos processed in each round of operation, and the corresponding grouping mode.
- each group of slave processing circuits comprises Rs slave processing circuits
- the master processing circuit is further configured to divide among the Rs slave processing circuits as follows Input feature map:
- the output feature map is evenly divided into Rs output feature blocks of the same shape in the HW dimension;
- the input feature map is divided into Rs input feature blocks in the HW dimension, so as to be allocated to the Rs slave processing circuits.
- Clause D5. The computing device of clause D4, wherein said computing device further comprises a first storage circuit and a second storage circuit,
- the convolution kernel is determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so as to be transmitted to the scheduled multiple slave processing circuits through the broadcast bus during operation;
- the input feature map is determined as distribution data, and the distribution data after splitting and transforming the dimension storage sequence is stored in the second storage circuit, so as to be distributed to corresponding slave processing circuits.
- Clause D6 The computing device according to Clause D5, wherein the Rs input feature blocks are respectively split according to the splitting unit and stored in the second storage circuit after being converted into the order of dimension storage as the Rs from the memory area allocated by the processing circuit.
- each said slave processing circuit comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:
- the first buffer circuit is used for buffering a plurality of input feature data rows from the second storage circuit distributed to the slave processing circuit;
- the second buffer circuit is configured to buffer a plurality of weight data rows from the first storage circuit that are multicast transmitted to the convolution kernel of the corresponding output channel value of the slave processing circuit;
- Each operation circuit is used to perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the weight data row selected from the second buffer circuit in each operation.
- Each calculation circuit calculates a plurality of output points spaced in X and/or Y dimensions on the output feature map for each calculation.
- Clause D9 The computing device of Clause D8, wherein said convolution operation is a three-dimensional convolution operation, and each of said slave processing circuits is further configured to:
- Kx and Ky are respectively the size of the convolution kernel in the X and Y dimensions or the slave processing circuit in the convolution The smaller value among the maximum kernel sizes supported by a single operation in split mode.
- each said arithmetic circuit is further configured to:
- the Nk*Nop parts calculated in the Nk sliding calculations are accumulated according to the corresponding convolution output points to obtain Nop operation results.
- each said slave processing circuit is further configured to:
- the partial operation results of the internal partial operation circuit are output each time, and the partial operation results are continuous on the X and/or Y dimensions of the output feature map.
- Clause D15 The computing device of Clause D8, wherein at each computation, each arithmetic circuit computes 2x2 output points spaced apart by 1 in both X and Y dimensions.
- Clause D17 The computing device according to Clause D9, wherein the maximum convolution kernel size supported by a single operation of the slave processing circuit in the convolution split mode is 8 ⁇ 8.
- Clause D18 A chip comprising the computing device according to any one of clauses D1-D17.
- Clause D20 A method of performing a convolution operation using the computing device of any one of Clauses D1-D17.
- a computing device configured to perform a depthwise convolution operation, the computing device comprising:
- a main processing circuit the main processing circuit is used to: obtain an input feature map and/or a convolution kernel, wherein the input feature map and the convolution kernel have been split into multiple split units according to the convolution split scheme and Convert dimension storage order, where a split unit includes data of the lowest storage dimension and at least one other storage dimension, and the total data volume of a split unit does not exceed the maximum single operation of the hardware, and the data in a split unit is continuous stored as a data row; and
- a plurality of slave processing circuits are used to perform depth convolution operations on the input feature map and corresponding data rows of the convolution kernel.
- Clause E3 The computing device according to clause E2, wherein said convolution splitting scheme further indicates the number of calculation rounds in which said convolution operation is performed, the number Nc of C processed in each round of operation, and the corresponding grouping mode, wherein Nc is aligned to Uc.
- each group of slave processing circuits comprises Rs slave processing circuits
- said master processing circuit is further configured to divide among said Rs slave processing circuits as follows Input feature map:
- the output feature map is evenly divided into Rs output feature blocks of the same shape in the HW dimension;
- the input feature map is divided into Rs input feature blocks in the HW dimension, so as to be distributed to the Rs slave processing circuits.
- Clause E6 The computing device of clause E5, wherein said computing device further comprises a first storage circuit and a second storage circuit,
- the convolution kernel is determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so as to be transmitted to the scheduled multiple slave processing circuits through the broadcast bus during operation;
- the input feature map is determined as distribution data, and the distribution data after splitting and transforming the dimension storage sequence is stored in the second storage circuit, so as to be distributed to corresponding slave processing circuits.
- the Rs input feature blocks are respectively split according to the splitting unit and stored in the storage area allocated for the Rs slave processing circuits in the second storage circuit after being converted to a storage order of dimensions.
- each of said slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:
- the first buffer circuit is used for buffering a plurality of input feature data rows from the second storage circuit distributed to the slave processing circuit;
- the second buffer circuit is configured to buffer a plurality of weight data rows from the first storage circuit that are multicast transmitted to the slave processing circuit;
- Each operation circuit is used to perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the weight data row selected from the second buffer circuit in each operation.
- each computing circuit calculates 1 output point on the output feature map at intervals of X and/or Y dimensions.
- each operation circuit calculates different output points on the output feature map in X and/or Y dimensions.
- each said slave processing circuit is further configured to:
- Nk Kx*Ky, Kx and Ky are respectively the dimensions of the convolution kernel in the X and Y dimensions or the slave processing circuit in the convolution split.
- each said arithmetic circuit is further configured to:
- Nk*Uc output points calculated in the Nk sliding calculations are spliced according to the division method of the output points to obtain Nk*N CU calculation results on the Uc channels.
- each said slave processing circuit is further configured to:
- the partial operation results of the internal partial operation circuit are output each time, and the partial operation results are continuous on the X and/or Y dimensions of the output feature map.
- Clause E16 The computing device according to Clause E10, wherein the maximum convolution kernel size supported by a single operation of the slave processing circuit in the convolution split mode is 4 ⁇ 4.
- Clause E17 A chip comprising the computing device according to any one of clauses E1-E16.
- Clause E19 A method of performing a convolution operation using the computing device of any one of clauses E1-E16.
- a computing device configured to perform a deep convolution operation in reverse training of a neural network model, said computing device comprising:
- a main processing circuit the main processing circuit is used to: acquire input neuron data and/or neuron gradient data, wherein the input neuron data and neuron gradient data have been split into multiple Split the unit and convert the dimension storage order, where one split unit includes the data of the lowest storage dimension and at least one other storage dimension, and the total data volume of one split unit does not exceed the maximum single operation of the hardware, and one split unit
- the data within is stored contiguously as a data row;
- the initial lowest storage dimension of data and neuron gradient data, the size on channel C, Ux and Uy are the dimensions of the split unit on the initial X and Y storage dimensions of the input neuron data and neuron gradient data respectively
- M is the maximum calculation amount of the hardware at one time
- Ux Uy ⁇ Uc>1
- Uc M/4 n ,
- Clause F3 The computing device of clause F2, wherein the convolution splitting scheme further indicates a grouping manner for performing the depthwise convolution operation, wherein the grouping manner is for the input neuron data and the neuron Gradient data is sequentially divided into Ns schedulable slave processing circuits in units of Uc according to the channel C dimension, and each slave processing circuit processes input neuron data and neuron gradient data of different consecutive Uc C values.
- Clause F4 The computing device of clause F3, wherein said computing device further comprises a first storage circuit and a second storage circuit,
- the neuron gradient data is determined as unicast data, and the unicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so that the neuron gradients corresponding to different Uc C values can be transmitted through the broadcast bus during operation.
- the data are respectively transmitted to the scheduled Ns slave processing circuits;
- the input neuron data is determined as the distribution data, and the distribution data after splitting and converting the dimension storage order is respectively stored in the second storage circuit in the second storage circuit in a sequentially divided manner according to the dimension of the channel C and in units of Uc. In the storage area corresponding to the processing circuit, so as to distribute to the corresponding slave processing circuit.
- each of said slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:
- the first buffer circuit is used for buffering a plurality of input neuron data rows from the second storage circuit distributed to the slave processing circuit;
- the second buffer circuit is configured to buffer a plurality of neuron gradient data rows from the first storage circuit that are unicast transmitted to the slave processing circuit;
- Each operation circuit is configured to perform bitwise multiply-accumulate on the input neuron data row selected from the first buffer circuit and the neuron gradient data row selected from the second buffer circuit in each operation operation.
- each operation circuit calculates an output point adjacent to the X and/or Y dimension on the XY plane of the Uc channel C values of the weight gradient data;
- each operation circuit calculates different output points of the weight gradient data on the X and/or Y dimensions.
- each said slave processing circuit is further configured to:
- Nk ceil(Kx/2)*ceil(Ky/2)
- Kx and Ky are respectively the size of the weight gradient data in the X and Y dimensions or the The smaller value of the maximum weight gradient size supported by a single operation of the slave processing circuit in the convolution split mode.
- each said arithmetic circuit is further configured to:
- the input neurons corresponding to the same channel value will be in units of 1/Uc data rows
- the metadata and the neuron gradient data are multiplied and accumulated to obtain an output point at the same position on the XY plane of Uc;
- Nk output points at intervals in the X and/or Y dimensions on the Uc XY planes are calculated.
- each said slave processing circuit is further configured to:
- Clause F10 The computing device of Clause F9, wherein said main processing circuit is further configured to:
- Clause F13 The computing device according to Clause F7, wherein the maximum weight gradient size supported by a single operation of the slave processing circuit in the convolution split mode is 4 ⁇ 4.
- Clause F14 A chip comprising the computing device according to any one of clauses F1-F13.
- Clause F15 A board comprising the chip according to Clause F14.
- Clause F16 A method of performing a convolution operation using the computing device of any one of Clauses F1-F13.
- a computing device configured to perform a cross-product convolution operation in reverse training of a neural network model, said computing device comprising:
- a main processing circuit the main processing circuit is used to: acquire input neuron data and/or neuron gradient data, wherein the input neuron data and neuron gradient data have been split into multiple Split unit, where one split unit includes data in the lowest storage dimension and at least one other storage dimension, and the total data volume of a split unit does not exceed the maximum single operation of the hardware, and the data in a split unit is stored continuously is a row of data; and
- a plurality of slave processing circuits for performing the cross-product convolution operation on corresponding data rows of the input neuron data and neuron gradient data.
- the initial minimum storage dimension of the data, the size on the input channel Ci and the initial minimum storage dimension of the neuron gradient data, and the size on the output channel Co, Ux and Uy are respectively the split unit in the input neuron
- Clause G3 The computing device according to clause G2, wherein the convolution splitting scheme further indicates the number of operation rounds in which the depthwise convolution operation is performed, the number Nco of output channels Co processed in each round of operation, and the corresponding grouping mode, where Nco is aligned to Uc.
- Clause G6 The computing device of clause G5, wherein said computing device further comprises a first storage circuit and a second storage circuit,
- the neuron gradient data is determined as multicast data, and the multicast data after splitting and converting the dimension storage order is stored in the first storage circuit, so that the neuron gradients corresponding to different Uc Co values are transmitted through the broadcast bus during operation.
- the data are respectively transmitted to the scheduled N slave processing circuit groups, and each slave processing circuit group shares the same neuron gradient data of Uc and Co values;
- the input neuron data is determined as distribution data, and the distribution data after splitting and converting the dimension storage order is copied into N copies, and each copy is divided into Rs data blocks according to the direction of Ci and in units of Uc, and stored separately In the corresponding storage area in the second storage circuit, so as to distribute to the corresponding slave processing circuit.
- each of said slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:
- the first buffer circuit is used for buffering a plurality of input neuron data rows from the second storage circuit distributed to the slave processing circuit;
- the second buffer circuit is configured to buffer a plurality of rows of neuron gradient data multicasted to the slave processing circuit from the first storage circuit;
- Each operation circuit is configured to perform bitwise multiply-accumulate on the input neuron data row selected from the first buffer circuit and the neuron gradient data row selected from the second buffer circuit in each operation operation.
- each computing circuit calculates one output point of the weight gradient data on different Co, on consecutive Uc Ci, and on the same position in the XY dimension;
- each operation circuit calculates different output points of the weight gradient data on the X and/or Y dimensions.
- each said slave processing circuit is further configured to:
- Nk Kx*Ky, Kx and Ky are respectively the size of the weight gradient data in the X and Y dimensions or the slave processing circuit in the convolution The smaller value of the maximum weight gradient size supported by a single operation in split mode.
- each said slave processing circuit is further configured to:
- Nk Kx*Ky, Kx and Ky are respectively the size of the weight gradient data in the X and Y dimensions or the slave processing circuit in the convolution The smaller value of the maximum weight gradient size supported by a single operation in split mode.
- each said arithmetic circuit is further configured to:
- the unit of 1/Uc data rows will correspond to the same input channel Ci value
- the input neuron data and the neuron gradient data are multiplied and accumulated to obtain Uc output points of the assigned Co value in the Ci dimension;
- Nk*Uc output points are calculated, which are Nk output points continuous in X and/or Y dimensions on the XY plane on a single Co and Uc Ci.
- each said slave processing circuit is further configured to:
- Clause G13 The computing device of Clause G12, wherein said main processing circuit is further configured to:
- Clause G16 The computing device according to any one of clauses G9-G10, wherein the maximum weight gradient size supported by a single operation of the slave processing circuit in the convolution split mode is 4 ⁇ 4.
- Clause G17 A chip comprising the computing device according to any one of clauses G1-G16.
- Clause G19 A method of performing a convolution operation using the computing device of any one of Clauses G1-G16.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Computer Hardware Design (AREA)
- Neurology (AREA)
- Complex Calculations (AREA)
Abstract
本披露公开了一种计算装置、利用计算装置实施卷积运算的方法及相关产品。该计算装置可以包括在组合处理装置中,该组合处理装置还可以包括接口装置和其他处理装置。该计算装置与其他处理装置进行交互,共同完成用户指定的计算操作。组合处理装置还可以包括存储装置,该存储装置分别与计算装置和其他处理装置连接,用于存储该计算装置和其他处理装置的数据。本披露的方案对卷积运算进行优化,提高了运算处理效率。
Description
相关申请的交叉引用
本公开要求于2021年9月26日申请的、申请号为202111131388.5、发明名称为“计算装置、利用计算装置实施卷积运算的方法及相关产品”的中国专利申请的优先权。
本披露一般地涉及数据处理领域。更具体地,本披露涉及一种配置用于执行卷积运算的计算装置、利用计算装置实施卷积运算的方法、芯片和板卡。
目前,深度学习(Deep Learning)已经成为机器学习中的重要分支,也大力助推着人工智能(AI)的发展。深度学习的核心技术——深度神经网络(DNN)已在诸多行业有着广泛的应用。
神经网络是人工智能、深度学习中最为关键的技术之一,其中卷积神经网络(Convolution Neural Network,CNN)是最为重要的一种网络类型。卷积神经网络中最为关键的计算即为卷积层(Conv layer)的卷积运算(Convolution Operation)。卷积层的功能是对输入数据进行特征提取,通过多层卷积,能够抽取复杂特征,以保证网络具有足够的表达能力和泛化能力。神经网络模型中包含了大量的、各种类型的卷积运算,卷积运算的计算性能极大地影响整个神经网络模型的计算性能。当神经网络模型应用于不同领域时,例如语音识别、机器翻译、图像处理等等,其对应的输入特征图和权值的各个维度大小可能各有不同。为了充分利用深度学习处理器的硬件优势,需要针对不同规模的、不同类型的卷积运算进行优化,以提高执行神经网络模型的计算性能。
发明内容
为了至少解决如上所提到的一个或多个技术问题,本披露在多个方面中提出了一种计算装置,其通过对输入特征图和权值进行分块处理,可以使得各种维度尺寸的数据能够适配卷积运算的硬件,从而提高卷积运算的计算效率。本披露实施例的卷积运算可以是各种神经网络模型中的运算,这些神经网络模型可以应用于各种领域,诸如图像处理、语音处理、文本处理等等,这些处理例如可以包括但不限于识别和分类。
在第一方面中,本披露实施例提供了一种计算装置,配置用于执行卷积运算,所述计算装置包括主处理电路,所述主处理电路用于:获取输入特征图和/或卷积核,其中所述输入特征图和卷积核已分别按卷积拆分方案拆分成多个拆分单元并转换其维度存储顺序,其中所述卷积拆分方案是根据所述输入特征图拆分前的最低存储维度的大小确定的,所述卷积拆分方案指示所述拆分单元的形状,一个拆分单元包含的数据量不超过硬件单次最大运算量,并且一个拆分单元内的数据连续存储为一个数据行;以及多个从处理电路,所述多个从处理电路用于对所述输入特征图和卷积核的对应拆分单元执行卷积运算。
在第二方面中,本披露实施例提供了一种芯片,其包括前述第一方面任一实施例的计算装置。
在第三方面中,本披露实施例提供了一种板卡,其包括前述第二方面的任一实施例的芯片。
在第四方面中,本披露实施例提供了一种由前述第一方面任一实施例的计算装置实施卷积运算的方法。
通过如上所提供的计算装置、芯片、板卡以及由计算装置实施卷积运算的方法,本披露实施例的方案针对不同维度尺寸的输入特征图应用不同的卷积拆分方案,以适应硬件运算装置的处理能力,从而充分利用多个从处理电路的并行处理能力,可以有效提高卷积运算的运算效率。此外,在一些实施例中,输入特征图和权值可以通过不同的数据路径进行传输,从而支持输入特征图和权值的多种复用方式,进一步优化卷积运算,减小数据访存量。
通过参考附图阅读下文的详细描述,本公开示例性实施方式的上述以及其他目的、特征和优 点将变得易于理解。在附图中,以示例性而非限制性的方式示出了本公开的若干实施方式,并且相同或对应的标号表示相同或对应的部分,其中:
图1示出本披露实施例的板卡的结构图;
图2示出本披露实施例的组合处理装置的结构图;
图3示出本披露实施例的单核或多核计算装置的处理器核的内部结构示意图;
图4a-图4c示出可以应用本披露实施例的几种示例性卷积运算原理示例;
图5示出了根据本披露实施例的计算装置的示意性结构框图;
图6示出了根据本披露实施例的一种示例性数据存储顺序;
图7a-7d示出了根据本披露实施例的几种示例性分组模式;
图8示出了根据本披露实施例的输入特征图的示例性拆分示意图;
图9a-9d示出了根据本披露实施例的第二存储电路中的数据存储示意图;
图10a-10b示出了根据本披露实施例的运算电路的输出点划分示意图;
图11示出根据本披露实施例的Forward16方案的拆分和存储示意图;
图12示出根据本披露实施例的Forward16方案中的单次运算示意图;
图13示出根据本披露实施例的Forward16方案中的滑动卷积示意图;
图14示出根据本披露实施例的Forward16方案中滑动卷积结果的累加示意图;
图15示出根据本披露实施例的Forward16拆分方案的输出数据格式示意图;
图16示出根据本披露实施例的Forward4方案的拆分和存储示意图;
图17示出根据本披露实施例的Forward4方案中的单次运算示意图;
图18示出根据本披露实施例的Forward4方案中的滑动卷积示意图;
图19示出根据本披露实施例Forward4方案的输出数据格式示意图;
图20示出根据本披露实施例的Forward1方案中运算电路的输出点划分示意图;
图21示出根据本披露实施例的Forward1方案中的单次运算示意图;
图22示出根据本披露实施例的Forward1方案中的滑动卷积示意图;
图23示出根据本披露实施例Forward1方案的输出数据格式示意图;
图24示出根据本披露实施例的Update1方案中第二存储电路中的数据存储示意图;
图25示出根据本披露实施例的Update1方案中的滑动卷积示意图;
图26示出根据本披露实施例Update1方案的输出数据格式示意图;
图27a-图27b示出根据本披露实施例的Update4方案中不同分组模式下第二存储电路中的示例性存储内容;
图28示出根据本披露实施例的Update4方案中的单次运算过程示意图;
图29示出根据本披露实施例的Update4方案中的滑动卷积过程示意图;以及
图30示出根据本披露实施例的Update4方案的输出数据格式示意图。
下面将结合本披露实施例中的附图,对本披露实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本披露一部分实施例,而不是全部的实施例。基于本披露中的实施例,本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本披露保护的范围。
应当理解,本披露的权利要求、说明书及附图中可能出现的术语“第一”、“第二”、“第三”和“第四”等是用于区别不同对象,而不是用于描述特定顺序。本披露的说明书和权利要求书中使用的术语“包括”和“包含”指示所描述特征、整体、步骤、操作、元素和/或组件的存在,但并不排除一个或多个其它特征、整体、步骤、操作、元素、组件和/或其集合的存在或添加。
还应当理解,在本披露说明书中所使用的术语仅仅是出于描述特定实施例的目的,而并不意在限定本披露。如在本披露说明书和权利要求书中所使用的那样,除非上下文清楚地指明其它情况,否则单数形式的“一”、“一个”及“该”意在包括复数形式。还应当进一步理解,在本披 露说明书和权利要求书中使用的术语“和/或”是指相关联列出的项中的一个或多个的任何组合以及所有可能组合,并且包括这些组合。
如在本说明书和权利要求书中所使用的那样,术语“如果”可以依据上下文被解释为“当...时”或“一旦”或“响应于确定”或“响应于检测到”。
示例性硬件环境
图1示出本披露实施例的一种板卡10的结构示意图。如图1所示,板卡10包括芯片101,其是一种***级芯片(System on Chip,SoC),或称片上***,集成有一个或多个组合处理装置,组合处理装置是一种人工智能运算单元,用以支持各类深度学***台的存储能力和计算能力有很高的要求,此实施例的板卡10适用在云端智能应用,具有庞大的片外存储、片上存储和强大的计算能力。
芯片101通过对外接口装置102与外部设备103相连接。外部设备103例如是服务器、计算机、摄像头、显示器、鼠标、键盘、网卡或wifi接口等。待处理的数据可以由外部设备103通过对外接口装置102传递至芯片101。芯片101的计算结果可以经由对外接口装置102传送回外部设备103。根据不同的应用场景,对外接口装置102可以具有不同的接口形式,例如PCIe接口等。
板卡10还包括用于存储数据的存储器件104,其包括一个或多个存储单元105。存储器件104通过总线与控制器件106和芯片101进行连接和数据传输。板卡10中的控制器件106配置用于对芯片101的状态进行调控。为此,在一个应用场景中,控制器件106可以包括单片机(Micro Controller Unit,MCU)。
图2是示出此实施例的芯片101中的组合处理装置的结构图。如图2中所示,组合处理装置20包括计算装置201、接口装置202、处理装置203和存储装置204。
计算装置201配置成执行用户指定的操作,主要实现为单核智能处理器或者多核智能处理器,用以执行深度学习或机器学习的计算,其可以通过接口装置202与处理装置203进行交互,以共同完成用户指定的操作。
接口装置202用于在计算装置201与处理装置203间传输数据和控制指令。例如,计算装置201可以经由接口装置202从处理装置203中获取输入数据,写入计算装置201片上的存储装置。进一步,计算装置201可以经由接口装置202从处理装置203中获取控制指令,写入计算装置201片上的控制缓存中。替代地或可选地,接口装置202也可以读取计算装置201的存储装置中的数据并传输给处理装置203。
处理装置203作为通用的处理装置,执行包括但不限于数据搬运、对计算装置201的开启和/或停止等基本控制。根据实现方式的不同,处理装置203可以是中央处理器(central processing unit,CPU)、图形处理器(graphics processing unit,GPU)或其他通用和/或专用处理器中的一种或多种类型的处理器,这些处理器包括但不限于数字信号处理器(digital signal processor,DSP)、专用集成电路(application specific integrated circuit,ASIC)、现场可编程门阵列(field-programmable gate array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等,并且其数目可以根据实际需要来确定。如前所述,仅就本披露的计算装置201而言,其可以视为具有单核结构或者同构多核结构。然而,当将计算装置201和处理装置203整合共同考虑时,二者视为形成异构多核结构。
存储装置204用以存储待处理的数据,其可以是DRAM,为DDR内存,大小通常为16G或更大,用于保存计算装置201和/或处理装置203的数据。
图3示出了计算装置201为单核或多核装置时处理核的内部结构示意图。计算装置301用以处理计算机视觉、语音、自然语言、数据挖掘等输入数据,计算装置301包括三大模块:控制模块31、运算模块32及存储模块33。
控制模块31用以协调并控制运算模块32和存储模块33的工作,以完成深度学习的任务, 其包括取指单元(instruction fetch unit,IFU)311及指令译码单元(instruction decode unit,IDU)312。取指单元311用以获取来自处理装置203的指令,指令译码单元312则将获取的指令进行译码,并将译码结果作为控制信息发送给运算模块32和存储模块33。
运算模块32包括向量运算单元321及矩阵运算单元322。向量运算单元321用以执行向量运算,可支持向量乘、加、非线性变换等复杂运算;矩阵运算单元322负责深度学习算法的核心计算,即矩阵乘及卷积。
存储模块33用来存储或搬运相关数据,包括神经元存储单元(neuron RAM,NRAM)331、权值存储单元(weight RAM,WRAM)332、直接内存访问模块(direct memory access,DMA)333。NRAM 331用以存储输入神经元、输出神经元和计算后的中间结果;WRAM 332则用以存储深度学习网络的卷积核,即权值;DMA 333通过总线34连接DRAM 204,负责计算装置301与DRAM 204间的数据搬运。
示例性卷积运算类型
基于前述硬件环境,在一个方面中,本披露实施例提供了一种计算装置,其配置用于执行卷积运算,从而可以对例如神经网络模型中的卷积运算进行优化。神经网络模型中的卷积层可以执行卷积运算,通过对输入特征图(也称为输入数据、神经元或输入神经元)应用卷积核(也称为过滤器、权值等)做卷积处理,从而进行特征提取。卷积层内部可以包含多个卷积核,组成卷积核的每个元素对应一个权重系数和一个偏差量bias。
神经网络模型中可能包含各种卷积运算层,例如执行正向、常规3D卷积运算的卷积层、执行深度(Depthwise)卷积运算的反卷积层。而在反向训练中,可能需要执行反向的深度卷积运算或叉乘卷积运算。本披露实施例可以针对这些不同类型的卷积运算进行优化。
在常规3D卷积运算中,假设卷积层中输入特征图(Feature map)张量形状表示为X[N Hi Wi Ci],卷积核(kernel)的张量形状表示为K[Co Kh Kw Ci],输出的结果为Y[N Ho Wo Co],那么,简化的卷积运算的数学计算公式可以表示如下:
Y
in,jc,jh,jw=∑
0≤ic≤ci,0≤ih≤kh,0≤iw≤kwX
in,ic,jh×sh+ih,jw×sw+iw×K
jc,ic,ih,iw (1)
上式中,X是输入数据,Y是输出数据,K是卷积核,Kh和Kw是K的长和宽,sh和sw是在长和宽方向上的步长(stride),公式忽略了偏差量bias,填充pad和膨胀dilation,并且假设输入数据X已经做了填充,卷积核已经做了膨胀。公式忽略了N维度和C维度,神经网络模型的正向计算在N维度上的计算都是独立的,在C维度上是全连接的。卷积核在工作时,会按照一定的步长扫过输入特征,在卷积窗口内对输入特征做矩阵元素乘法求和并叠加偏差量。在常规3D卷积运算中,H、W和Ci方向的对位乘积结果会进行累加,因此称为3D卷积。但是这种3D卷积存在约束条件:卷积核的Ci维度大小和输入特征图的Ci维度大小相等,因此卷积核不在Ci方向滑动,是一种伪3D卷积。为了区别于本文中其他卷积运算,上述卷积运算称为3D卷积运算。
图4a示出了可以应用本披露实施例的示例性常规3D卷积运算原理示例。
图中示例性示出了大小为[N Hi Wi Ci]的四维输入数据X,其可以表示成N个Hi×Wi×Ci大小的立体矩形410a。图中还示例性示出了大小为[Co Kh Kw Ci]的四维卷积核K,其可以表示成Co个Kh×Kw×Ci大小的立体卷积核420a。输入数据X与卷积核K的卷积结果得到输出数据Y,其为[N Ho Wo Co]大小的四维数据,可以表示成N个Ho×Wo×Co大小的立体矩形430a。
图中还具体示出了一个卷积运算示例,其中输入数据为6×6×3大小的输入特征图440a,省去N维度;卷积核为3×3×3大小的立体卷积核450a,针对单个卷积核Co;输出数据为4×4的输出特征图460a。具体运算过程如下:
卷积核450a按照一定的步长扫过输入特征图440a,在卷积窗口470a内对输入特征做矩阵元素乘法求和并叠加偏差量。也即,输出特征图460a中每个位置上的值由每个输入特征图的对应区块和对应卷积核做二维卷积运算之后再加和得到。例如,图中示出了输出特征图460a上(0,0)位置的值(也即卷积输出点)由输入特征图中黑色立方体框出的卷积窗口470a与立体卷积核450a 进行二维卷积运算得到3个值,再加和得到最终值。
为了得到其他位置的输出,可以在输入特征图440a上移动卷积核450a的位置,也即移动卷积输出点的卷积窗口。在图中示例中,卷积步长(Sx,Sy)为(1,1),当横向(宽度方向)向右或纵向(高度方向)向下移动一格后做卷积运算,可以分别得到输出特征图460a上(0,1)或(1,0)位置的值。
从上面的描述可知,在神经网络的一个卷积层中,有N组输入特征图,每组包含Hi×Wi×Ci个信息,其中Hi和Wi分别是输入特征图的高度和宽度,Ci是输入特征图的个数,也称为输入通道数。卷积层有Ci×Co个Kh×Kw大小的卷积核,其中Ci是输入通道数,Co是输出特征图的个数(或输出通道数),Kh和Kw分别是卷积核的高度和宽度。输出特征图包含Ho×Wo×Co个信息,其中Ho和Wo分别是输出特征图的高度和宽度,Co是输出通道数。此外,在卷积运算中,还会涉及到卷积步长(Sx,Sy),卷积步长的大小会影响输出特征图的尺寸。
图4b示出了可以应用本披露实施例的示例性深度卷积运算原理示例。
深度卷积与常规3D卷积相比,区别在于深度方向不进行累加,此处的深度方向是指输入通道Ci。在常规3D卷积中每个卷积核都需要与输入特征图的所有图层(输入通道)计算并累加,所以每个卷积核的输入通道数等于输入特征图的输入通道数。而深度卷积中每个卷积核都是单通道的,一个卷积核负责一个通道,一个通道只被一个卷积核卷积。因此,深度卷积有时候也称为2D卷积,也即只在H和W维度上滑动累加。
如图所示,输入特征图410b的维度尺寸为12×12×3,也即包括三个通道,每个通道包括12×12的图像。在此深度卷积中分别使用3个卷积核420b,每个卷积核都是单通道的,其尺寸例如为5×5×1。每个卷积核仅对输入特征图410b的一个通道做卷积,这样的卷积每次都得出大小为8×8×1的输出,之后再将这些输出堆叠在一起创建一个8×8×3的图像,最终得出一个大小为8×8×3的输出特征图430b。从图中可以看出,输出特征图的深度(通道数)保持与输入特征图一致。
由于在深度卷积中输入通道不进行累加,因此在涉及深度卷积时,其输入特征图、卷积核和输出特征图的维度都可以简化成C(通道)、H(高度)、W(宽度)三个维度。
在神经网络模型训练的反向传播中,涉及到神经元梯度和权值梯度的计算,如下所示:
其中,top_diff、bottom_diff分别为神经元梯度,W是本次迭代的权值,△W是本次迭代计算的权值梯度,
是反向传播中的计算,类似于卷积运算。相对于反向传播方向上,前一层的bottom_diff是当前层的top_diff,当前层的bottom_diff是下一层的top_diff,由此可以反向逐层传递误差。
在公式(2)的计算中,top_diff与W之间的运算类似于输入神经元和权值W之间的运算过程,其中top_diff相当于输入特征图。
在公式(3)的计算中,top_diff与bottom_data之间的运算类似于深度卷积运算,其中top_diff相当于卷积核,在bottom_data的XY方向上滑动累加,其运算原理可以参考图4b。在此运算场景中,top_diff与bottom_data的尺寸通常都比较大。因此,本披露实施例针对此种场景下的卷积运算(简称反向深度卷积)也提供优化方案。
在反向传播中,对于执行常规3D卷积运算的卷积层,其反向过程中的运算可以称为叉乘卷积运算。本披露实施例同样可以针对这种卷积运算提供优化方案。
图4c示出了可以应用本披露实施例的叉乘卷积运算原理示例。
图中示例性示出了大小为[Ho Wo Co]的三维数据top_diff,其可以表示成Ho×Wo×Co大小的立体矩形410c;图中还示出了大小为[Hi Wi Ci]的三维数据bottom_data,其可以表示成Hi×Wi×Ci大小的立体矩形420c。top_diff与bottom_data执行叉乘卷积运算后得到输出数据430c,其是大小为[Co Kh Kw Ci]的四维数据,其可以表示成Co个Kh×Kw×Ci大小的立体矩形430c。与图4a对比可以看出,图4c的叉乘卷积相当于常规3D卷积的反向运算,也即通过输出特征图 (top_diff)和输入特征图(bottom_data),计算出卷积核。图4c中省去了N维度。
具体地,对于top_diff中每一个HoWo面的数据,也即针对每个Co值的HoWo面,复制出Ci份,得到Ho×Wo×Ci的数据440c。将该数据440c与bottom_data执行深度卷积运算(参考图4b的示意图),也即Ci方向不累加,从而得到输出460c,其为Kh×Kw×Ci大小的三维数据。针对每个HoWo面,重复该复制和深度卷积运算,即可得到Co个Kh×Kw×Ci大小的三维数据,也即得到四维的卷积核430c,其大小为Co×Kh×Kw×Ci。
在本文中,输入特征图(Feature map)、输入数据、神经元或输入神经元可互换使用;卷积核、过滤器或权值可互换使用。此外,H(高度)和Y维度可互换使用,W(宽度)和X维度可互换使用。相应地,输入特征图的H维度可以表示为Hi或Yi,输出特征图的H维度可以表示为Ho或Yo,W维度类似表示。在本披露实施例中,每个卷积输出点具有对应的卷积窗口,卷积窗口的形状等于卷积核的形状。每个卷积输出点的值对应于其卷积窗口内的输入特征图与权值的对位乘累加结果。此外,无论涉及哪种类型的卷积运算,所涉及的数据都可以分为输入特征图、卷积核和输出特征图。例如,在反向运算中,top_diff对应卷积核,bottom_data对应输入特征图,△W对应输出特征图。
示例性计算装置
在本披露实施例中,可以采用主从结构的计算装置来实施上述卷积运算。进一步地,可以为输入特征图和卷积核配置不同的数据通路,从而提高访存效率。
图5示出了根据本披露实施例的计算装置500的示意性结构框图。可以理解,该结构可以视为图3中单个处理核的运算模块的内部结构细化,也可以视为在多个图3所示处理核的运算模块基础上联合的功能划分框图。如图5所示,本披露实施例的计算装置500可以配置用于执行各种类型的卷积运算,其可以包括主处理电路(MA)510和多个从处理电路(SL)520,图中示出了16个从处理电路SL0~SL15。本领域技术人员可以理解,从处理电路的数量可以更多或更少,取决于具体的硬件配置,本披露实施例在此方面没有限制。
主处理电路和从处理电路之间以及多个从处理电路之间可以通过各种连接相互通信。在不同的应用场景中,多个从处理电路之间的连接方式既可以是通过硬线布置的硬连接方式,也可以是根据例如微指令进行配置的逻辑连接方式,以形成多种从处理电路阵列的拓扑结构。本披露实施例在此方面没有限制。主处理电路和从处理电路可以相互配合,由此实现并行运算处理。
为了支持运算功能,主处理电路和从处理电路可以包括各种计算电路,例如可以包括向量运算单元及矩阵运算单元。向量运算单元用以执行向量运算,可支持向量乘、加、非线性变换等复杂运算;矩阵运算单元负责深度学习算法的核心计算,例如矩阵乘和卷积。
从处理电路例如可以用于根据运算指令,对相应的数据并行执行中间运算得到多个中间结果,并将多个中间结果传输回主处理电路。
通过将计算装置500设置成主从结构(例如一主多从结构,或者多主多从结构,本披露在此方面没有限制),对于正向运算的计算指令,可以根据计算指令将数据进行拆分,从而通过多个从处理电路对计算量较大的部分进行并行运算以提高运算速度,节省运算时间,进而降低功耗。
在本披露一些实施例中,通过利用不同的数据通路传输输入特征图和权值,可以支持输入特征图和权值的多种复用方式,从而减小运算期间的数据访存量,提升处理效率。
具体地,计算装置500中还可以包括第一存储电路530和第二存储电路540,用于分别存储经由不同数据通道传输的数据。可选地,该第一存储电路530和第二存储电路540可以是同一存储器划分形成的两个存储块,也可以是两个独立的存储器,此处不做具体限定。
第一存储电路530可以用于存储多播数据,也即第一存储电路中的数据将通过广播总线传输给多个从处理电路,这些从处理电路接收到相同的数据。可以理解,通过广播总线可以实现广播和多播。多播是指将一份数据传输到多个从处理电路的通信方式;而广播是将一份数据传输到所有从处理电路的通信方式,是多播的一个特例。由于多播和广播都对应一对多的传输方式,本文中未对二者特意区分,广播和多播可以统称为多播,本领域技术人员根据上下文可以明确其含义。
第二存储电路540可以用于存储分发数据,也即第二存储电路中的数据将分别传输给不同的从处理电路,每个从处理电路接收到不同的数据。
通过分别提供第一存储电路和第二存储电路,可以支持针对待运算的数据以不同传输方式进行传输,从而通过在多个从处理电路之间复用多播数据来降低数据访存量。
在一些实施例中,可以将输入特征图和卷积核中之一确定为多播数据并存储在第一存储电路中,以在运算期间通过广播方式将数据传输给调度的多个从处理电路。对应地,可以将输入特征图和卷积核中另一确定为分发数据并存储在第二存储电路中。这些分发数据可以在运算前分发给对应的从处理电路。
图5还示出了根据本披露实施例的从处理电路SL的内部结构示意图。如图所示,每个从处理电路520可以包括多个运算电路CU 521、第一缓冲电路522和第二缓冲电路523。图中示出了4个运算电路CU0~CU3。本领域技术人员可以理解,运算电路的数量可以更多或更少,取决于具体的硬件配置,本披露实施例在此方面没有限制。
在一些实施例中,第一缓冲电路522可以用于缓存分配给该从处理电路的权值或输入特征图。相应地,第二缓冲电路523则可以用于缓存分配给该从处理电路的输入特征图或权值。这两个缓冲电路均用于选取参与运算的数据。第一缓冲电路522的数据可以是来自例如第一存储电路530或第二存储电路540的多个数据行,对应地,第二缓冲电路523的数据可以来自例如第二存储电路540或第一存储电路530的多个数据行。取决于具体的复用方式,这些数据行可以在运算期间被分发给对应的运算电路CU 521或广播给该从处理电路520内的所有CU 521。
每个运算电路CU 521用于在每个运算周期内,针对分别从第一缓冲电路中选取的数据行和从第二缓冲电路中选取的数据行执行对位乘累加运算。
通过分别提供第一缓冲电路和第二缓冲电路,可以支持针对待运算的数据以不同传输方式进行传输,从而通过在单个从处理电路内的多个运算电路之间尽可能复用数据来降低数据访存量。
从处理电路520中还可以包括第三缓冲电路524,用于缓存各个运算电路CU 521的运算结果。
可以理解,虽然在图5中将各个处理电路与存储电路示出为分立的模块,但是根据不同的配置,存储电路与处理电路也可以合并成一个模块。例如,第一存储电路530可以与主处理电路510合并在一起,第二存储电路540则可以由多个从处理电路520共享,并为每个从处理电路分配独立的存储区域,加速访问。本披露实施例在此方面没有限制。此外,在该计算装置中,主处理电路和从处理电路可以属于同一处理器或芯片的不同模块,也可以属于不同处理器,本披露在此方面也没有限制。
示例性数据拆分和存储
在本披露实施例中,所涉及的多维数据的维度表征为(N,H,W,C)或(Co,H,W,Ci),其代表了数据在存储器中的存储顺序。可以理解,虽然多维数据具有多个维度,但是因为存储器的布局始终是一维的,因此多维数据与存储器上的存储顺序之间存在对应关系。多维数据通常被分配在连续的存储空间中,也即可以将多维数据进行一维展开,按顺序存储在存储器上。例如,在本披露实施例中,初始的输入特征图可以按照低维度(此处C/Ci为最低维度)优先方式,进行顺序存储;而为了优化卷积运算,在运算过程中或者在运算之前可以调整输入特征图的存储顺序,如后面将详细描述的。相邻的维度是指多维数据的维度信息表示中相互紧挨着的维度,例如,W和Ci相邻,相邻的维度也可以称为连续的维度。
在智能处理器中,出于算力的需要和面积功耗开销的考虑,硬件的主要运算单元是向量的乘加运算器。在硬件设计中实现各类卷积算法的支持,本质上是最大化地提取算法中的乘加运算,并且通过数据通路实现在片上RAM(诸如图3中的NRAM、WRAM等)和运算器之间高效地交换乘加运算的输入和输出数据。
硬件在存储上是以一行一行(缓存行)进行存储的,读、写、计算操作在整行对齐时效率最高,因此为了充分利用带宽,适配运算器阵列的访存量等需求,通常需要将数据进行向量化对齐。人工智能芯片的设计通常以Ci维度为最低维度,也即上述NHWC摆放顺序,Ci维度上的数据是 连续的。因此,向量化对齐要求需要Ci维度的大小对齐到指定数值,例如对齐值M,从而以该对齐值M为单位进行存取数,M也可以称为硬件单次最大运算量。基于不同的硬件设计,M可以有不同的数值,例如64bit、128bit、256bit、512bit等。通常,运算器阵列的输入端口大小也与M相关,例如在输入数据位宽对称的情形下,运算器阵列的输入端口大小通常为M的2倍,也即一次性处理对齐值M规模的输入特征图数据和权值数据。当输入特征图的Ci维度较大时,比较容易满足上述对齐要求。
当输入特征图的Ci维度较小时,例如小于一个缓存行的大小,则需将Ci维度补齐到一行数据(例如,512比特),即填充无效数据0。这种填充会造成大量的冗余计算,导致资源浪费,降低了运算的效率。
在本披露实施例中,提出了一种卷积运算方案,其例如可以由图5的计算装置来执行。其中,主处理电路用于获取输入特征图和/或卷积核,输入特征图和卷积核已分别按照卷积拆分方案拆分成多个拆分单元并转换维度存储顺序,以使得一个拆分单元内的数据连续存储为一个数据行。取决于不同的硬件配置和/或其他考虑因素,输入特征图和卷积核的上述拆分和维度转换可以在不同位置、不同时间执行。在反向传播的神经元梯度更新过程中,可以将top_diff看成是输入特征图。
在一些实施例中,主处理电路可以包括分块电路,也即分块电路集成在主处理电路中,用于分别针对输入特征图和卷积核进行拆分和维度转换存储。例如,主处理电路可以从外部存储电路(例如DDR)读取原始存储格式的输入特征图和卷积核,然后利用分块电路对输入特征图和卷积核分别进行拆分和维度转换,之后将输入特征图和卷积核之一存储在第一存储电路中,而将另一存储在第二存储电路中。上述拆分过程可以在运算期间执行也可以在运算前执行,以准备好数据。
在另一些实施例中,主处理电路可以包括部分分块电路,该部分分块电路用于仅针对输入特征图和卷积核中被确定为多播数据的数据进行拆分和维度转换存储,而被确定为分发数据的数据则可以通过外部分块电路进行拆分和维度转换。例如,在一个示例中,被确定为分发数据的卷积核可以通过外部电路进行拆分和维度转换后,预先存储在第二存储电路中,其可以从片外存储电路直接存储到第二存储电路中,也可以经由第一存储电路再存储到第二存储电路中。
在又一些实施例中,主处理电路可以完全不包括分块电路或不执行分块电路的功能。在这些实施例中,输入特征图和卷积核由独立于主处理电路的分块电路进行拆分和维度转换存储。经拆分和维度转换后的输入特征图和卷积核之一可以存储在第一存储电路中,而另一可以存储在第二存储电路中。
可以根据输入特征图的最低存储维度(例如Ci)的大小,确定对应的卷积拆分方案,其中卷积拆分方案至少指示待运算数据的拆分单元的形状。一个拆分单元包含的数据量不超过硬件单次最大运算量。
在一些实施例中,一个拆分单元包含的数据量可以设置成硬件的一次性处理对齐值M,从而以拆分单元为单位进行运算处理,可以充分发挥硬件的算力,避免或减少无效计算。
在本披露的示例性描述中,不防假设M=512bit=64Byte,数据类型可以是Int8、Int16、Float16或Float32,并且输入特征图与卷积核的数据类型一致。由于数据类型至少需要1字节的宽度,并且运算处理的最小单位是一个数据,因此在下面的示例中均以字节为单位进行各种计算,例如M=64B,Ci=28B等等,其中有时候为了简洁起见省略单位。
当拆分单元的数据量等于M时,每个拆分单元的数据块形状为blockC*blockY*blockX,其可能存在多种情形,表1列出了其中的几种:
表1、数据块形状
从表1可以看出,有些数据块形状的X和Y维度尺寸相等(如深色行所示),这种形状可以简化后续的运算。因此在本披露实施例中,可以优选使用这种数据块形状对待运算数据进行拆分。
为了简便起见,将64B×1×1形状的拆分方案称为Forward64,将16B×2×2形状的拆分方案称为Forward16,将4B×4×4形状的拆分方案称为Forward4,将4B×4×4形状的应用于深度卷积运算的拆分方案称为Forward1,将4B×4×4形状的应用于反向深度卷积运算的拆分方案称为Update1,将4B×4×4形状的应用于叉乘卷积运算的拆分方案称为Update4。除了Forward64之外,这些拆分方案适合卷积计算中通道C比较小的场景,因此也可以统称为小卷积。在这些小卷积拆分方案中,一个拆分单元包括最低存储维度和至少一个其他存储维度的数据,并且一个拆分单元的总数据量不超过硬件单次最大运算量。
不同的卷积拆分方案可以适用于不同的运算场景,从而获得不同程度的性能优化。具体地,在一些实施例中,可以按照如下至少一项规则来确定对应的卷积拆分方案:
存在多个最近的M/4
n的倍数时,取M/4
n中最大值作为Uci,或者取其中对齐填补量最小的M/4
n作为Uci;以及
确定拆分单元在X和Y存储维度上的大小Ux(也即blockX)和Uy(blockY),使得Uci×Uy×Ux=M,其中优选Ux=Uy。
以下结合几个示例来描述上述规则的应用。所有示例中假设M=64,则M/4
n可以是64、16和4。
在一个示例中,假设Ci=28,则对齐到最近的M/4
n的倍数是4*7,此时将拆分单元在最低存储维度上的大小Uc(也即blockC)确定为4。当优选Ux=Uy,则可以确定拆分单元的形状为4B×4×4,也即Forward4方案。
在另一个示例中,假设Ci=112,若对齐到64*2=128,补零16个;若对齐到16*7=112,无需补零;若对齐到4*28=112,也无需补零。此时,最近的M/4
n的倍数是16*7=4*28=112,根据规则,可以取M/4
n中最大值16作为Uc。当优选Ux=Uy,则可以确定拆分单元的形状为16B×2×2,也即Forward16方案。
在确定了拆分方案之后,接着可以按照所确定的卷积拆分方案,将输入特征图和卷积核拆分成多个对应的拆分单元并转换其维度存储顺序,以使得一个拆分单元内的数据连续存储为一个数据行,从而方便后续以拆分单元(数据行)为单位进行读取处理。
在一些实施例中,对于三维或者四维的神经元或者权值的数据,将其全部划分为大小为blockC*blockY*blockX(Uc×Uy×Ux)大小的数据块,每一个数据块连续存储在例如M=64B的一行上,由此在读取一行数据时,实际取出一个数据块的数据。
具体地,可以从以第一维度存储顺序存储的待运算数据中,以拆分单元为单位,按第一读取顺序读取一个或多个拆分单元,将读取的拆分单元存储到对应的存储电路上,其中每个拆分单元内的数据按照第二维度存储顺序存储,拆分单元之间按照第三维度存储顺序存储。
图6示出了根据本披露实施例的一种示例性数据存储顺序。
如图所示,610表示待运算的四维张量的存储方式,包含N个3维的子张量,N在最高维度,也即四维张量的第一维度存储顺序为NHWC。注意,本文中H和Y、W和X可互换使用。每一个子张量被划分为更小的数据块或拆分单元,每一维的数据块的个数分别为C/Y/X。
中间的图620表示每一个子张量的存储方式,每个数据块被存储为连续的64Byte,也即一行。当读取数据块的顺序不同时,行之间的顺序也会相应地变化。在图中示例中,按照先C、然后X、 最后Y的方向读取数据块,也即第一读取顺序为YXC,则各行之间按照Y*X*C的顺序存储,也即第三维度存储顺序为YXC或HWC。在此示例中,第三维度存储顺序与第一维度存储顺序相同。可以理解,也可以使用其他读取顺序,从而导致第三维度存储顺序与第一维度存储顺序不同,此处不再一一列举。
右侧的图630表示每一行内的顺序,也即每个数据块内的数据顺序,其形状为blockC*blockY*blockX,此时第二维度存储顺序为CYX或CHW。后面将结合各种示例性卷积拆分方案详细描述具体的拆分方案。
示例性分组运算和数据复用
前面描述了本披露实施例的计算装置的硬件结构以及数据的示例性拆分方案和存储方式,上述硬件结构可以为参与运算的输入特征图和权值提供不同的数据通路,从而利用不同的数据传输方式(例如,广播、多播、分发等)来减少运算期间的数据访存量,提高运算效率。卷积的计算是每一个输入特征图都需要和每一个Co的卷积核进行乘加运算,从而输出Co个输出特征图。然而,并不是片上空间一定能同时存储下所有规模的卷积核和输入特征图,因此,对于硬件而言存在一系列重复加载输入特征数据或者权值数据的操作,如何平衡重复加载输入特征数据还是权值数据对计算的效率会有一定影响。在实际运算中,为了减少频繁的片外访存,根据参与运算的数据的规模特性,可以采取不同的复用方式。在卷积运算中,主要有两种数据复用方式:卷积核复用和输入特征图复用。
根据复用的场景,卷积核复用又可以分为通道内卷积核复用和批次间卷积核复用。通道内卷积核复用是针对单个输出通道,也即一个输出特征图的情况,此时只有一组卷积核。对于每个输入特征图,多个卷积窗口可以复用相同的卷积核。批次间卷积核复用是针对批处理的情况,即对多个输入图像同时处理。多个输入图像采用相同的卷积核组进行处理,因此卷积核可以复用。
类似地,根据复用的场景,输入特征图复用可以分为通道内输入特征图复用和通道间输入特征图复用。通道内输入特征图复用是针对单个输出通道,对于每个输入特征图,其相邻的卷积窗口可以复用部分输入特征图的数据。通道间输入特征图复用是针对多个输出通道,也即有多个输出特征图(即多组卷积核)的情况,此时一个卷积窗口内的输入特征图可以与多组卷积核做卷积运算。
根据前面描述的卷积运算原理可知,Co维度(深度卷积为C维度)上的运算结果无需累加,因此不同Co上的运算分配在不同的运算电路上可以相对独立地进行。在输入通道数量较小的场景下,卷积核普遍也较小,例如Kh和Kw通常是个位数,Co与Ci的尺寸差不多。在这些实施例中,通常单轮运算中卷积核的输出通道Co维度的尺寸不超过所调度的从处理电路的数量,因此单个Co的运算需要由一个或多个从处理电路来完成。更一般地,即使Co维度较大时,也可以通过拆分成多轮运算来实现,其中每轮运算处理的Co尺寸不超过所调度的从处理电路的数量。由此,在一个示例中,可以首先基于卷积核的输出通道Co维度尺寸和可调度的从处理电路数量Ns,确定完成卷积运算所需的运算轮次以及各轮次运算中处理的Co数量或相应的分组模式。
在确定完成卷积运算所需的运算轮次时,各轮次所处理的Co数量可以不尽相同,从而即使对于同一Co维度尺寸也可能存在多种分配方式。
例如,以图5中示出的包括16个从处理电路SL的计算装置为例,假设所有从处理电路都可调度,也即Ns=16。当Co=40时,可以分成三轮运算,第一轮处理前16个Co值,每个SL处理一个不同的Co值;第二轮处理接下来的16个Co值,每个SL处理一个不同的Co值;最后一轮处理剩下的8个Co值,每2个SL处理一个不同的Co值。在另一种分配方式中,也可以分成两轮运算,第一轮处理前32个Co值,每个SL处理2个不同的Co值;最后一轮处理剩下的8个Co值,每2个SL处理一个不同的Co值。又例如,当Co=12时,可以分成单轮运算,每个SL处理一个不同的Co值,其中有4个SL空闲或进行无效运算。在另一种分配方式中,也可以分成三轮运算,每次处理4个连续的Co值,每4个SL处理1个不同的Co值,从而在每轮运算中利用所有可调度的从处理电路。可以理解,本领域技术人员还可以设想出更多其他分配方案。
由此可见,不管哪种分配方式,在单轮运算中,Co可能存在两种分配情况:多个从处理电路处理一个Co值,或者单个从处理电路处理一个或多个Co值。具体地,在处理Nco个输出通道的单个运算轮次中,每Rs个SL构成一个从处理电路组SLB,处理对应同一输出Co值的卷积核,Rs=[Ns/Nco],也即同一卷积核在同一SLB内的Rs个SL上复用,Rs表示卷积核在从处理电路之间的复用次数。与之相应地,输入特征图可以在各个从处理电路组SLB之间复用,Rn=[Ns/Rs],表示输入特征图在从处理电路之间的复用次数。
可选地或附加地,当每个从处理电路处理对应rn个Co值的卷积核,rn=[Nco/Ns],此时每个从处理电路处理的输入特征图可以重复用于rn个卷积核,rn表示输入特征图在单个从处理电路内的复用次数。可以考虑硬件缓冲空间限制等因素(例如图5中的第一缓冲电路和第二缓冲电路的大小)来确定单个从处理电路内可应用的最大卷积核复用次数rs和最大输入特征图复用次数rn。
考虑到硬件电路中的缓存大小限制和复用收益,在本披露一些实施例中暂时不考虑一个从处理电路在单轮运算中处理多个Co值的情况,而只考虑一个或多个从处理电路在单轮运算中只处理一个Co值的情况。
根据在单轮运算中处理同一Co值的从处理电路SL的个数,可以采用不同的分组模式。可以理解,优选对可调用的从处理电路SL平均分配,从而均衡算力,例如,每2个SL一组,从而16个SL可以同时处理8个Co值;或者每4个SL一组,从而16个SL可以同时处理4个Co值;等等。在一些实施例中,对于图5所示的包括Ns=16个SL的计算装置,可以选择如下几种分组模式:Group1模式、Group4模式和Group16模式。本领域技术人员可以理解,根据Ns的数值不同,可以有不同的分组模式,每种分组模式均可以参考本文给出的以上三种代表性分组模式进行对应的处理。
在一些实施例中,上述分组模式可以统一表示为GroupN,代表当前轮次运算中调度的所有从处理电路SL分为N组,每个从处理电路组SLB处理同一Co值,不同从处理电路组SLB处理不同Co值。对于总计16个SL可调度的场合下,N可以取1,4,16,分别对应上面的Group1、Group4和Group16。
图7a-7d示出了根据本披露实施例的几种示例性分组模式。图7a示出了Group1模式,图7b示出了Group16模式,图7c示出了一种Group4模式,以及图7d示出了另一种Group4模式。
如图7a所示,Group1模式是指所有可调度的16个SL属于一个组,共同处理一个Co值,例如SL0~SL15属于组G0。从而,针对该一个输出通道的运算被分配在16个SL上。在这种模式下,可以优先考虑将该输出通道的卷积核720以广播方式传输到各个SL,输入特征图710则进行拆分分配给各个SL,从而提高访存效率。
在一个实施例中,可以将卷积核存储在图5的第一存储电路530上,以利用广播通道进行传输。输入特征图则可以按照输出特征图的XY方向划分,存储在第二存储电路540上,以分配给不同的SL。由此,所有SL共同计算一个Co的输出特征图。后面将结合附图详细描述输入特征图的划分和存储。
如图7b所示,Group16模式是指所有可调度的16个SL分成16个组,也即每组一个SL,每个SL处理一个不同的Co值。例如SL0属于组G0,SL1属于组G1,以此类推,直至SL15属于组G15。在这种模式下,同一块输入特征图730可以在16个SL之间重复使用,因此可以优先考虑将输入特征图730以广播方式传输到各个SL,而对应不同Co的卷积核740则分发给对应的SL。
在一个实施例中,可以将输入特征图存储在图5的第一存储电路530上,以利用广播通道进行传输。卷积核则根据Co划分,存储在第二存储电路540上,以分配给不同的SL。由此,所有SL针对同一输入特征图计算不同Co的输出特征图。
Group4模式是指所有可调度的16个SL分成4个组,每组处理一个Co值。每个SL组(简称SLB)包括的SL数量等于Rs=Ns/4=4。例如SL0~SL3属于组G0,SL4~SL7属于组G1,SL8~SL11属于组G2,以及SL12~SL15属于组G3。这种模式介于Group1和Group16之间,因此可以将卷积核或输入特征图任一确定为多播数据,而将另一确定为分发数据。
在一个实施例中,可以将卷积核按照Co划分成4组,存储在图5的第一存储电路530上, 以利用广播通道进行传输。输入特征图则可以按照输出特征图的XY方向划分为4份并复制4份,存储在第二存储电路540上,以分发给4个SLB。每个SLB获得相同的输入特征图,在SLB内再按照所划分的4份分发给其内的4个SL。由此,每个SLB中的所有SL共同计算一个Co的输出特征图,4个SLB则分别处理一个不同的Co。
在另一实施例中,可以将卷积核存储在图5的第二存储电路540上,而将输入特征图存储在第一存储电路530上,划分方式与前一实施例类似。
在这种模式下,卷积核在SLB之间的划分又可以有多种方式。
图7c示出了一种卷积核的Co分配方式770。在此方式中,将卷积核分成4组,按照Co以间隔1为单位划分至各组。例如,当Co=12时,分成的4组Co分别为{0,4,8}、{1,5,9}、{2,6,10}和{3,7,11}。每一次发送各组的一个Co,例如第一次发送Co=0~3,一个Co对应一个SLB,在一个SLB内的4个SL共用相同权值;第二次发送Co=4~7,依次类推。由此,每轮运算完成后,各个SLB输出的运算结果的Co维度是连续的。
图7d示出了卷积核的另一种Co分配方式780。在此方式中,将卷积核按照Co连续地平均分成4组。例如,当Co=12时,分成的4组Co分别为{0,1,2}、{3,4,5}、{6,7,8}和{9,10,11}。每一次发送各组的一个Co,例如第一次发送Co=0,3,6,9,一个Co对应一个SLB,在一个SLB内的4个SL共用相同权值;第二次发送Co=1,4,7,10,依次类推。由此,每个SLB在多轮运算中输出的运算结果的Co维度是连续的。
输入特征图的示例性拆分
从前面的描述可以看出,当多个SL共同处理一个Co值时,需要在这多个SL之间对输入特征图进行拆分,例如Group1分组模式需要将输入特征图拆分成16份,而Group4分组模式需要将输入特征图拆分成4份。
为了保证拆分的输入特征图可以共用卷积核,可以根据输出特征图的Ho/Wo方向来划分,从而映射回到输入特征图的划分。在一些实施例中,在每个从处理电路组内包括的Rs个从处理电路SL之间可以按如下划分输入特征图:根据对应的输出特征图的尺寸,将输出特征图在XY维度(也即Ho/Wo维度)上平均划分为Rs个形状相同的输出特征块;以及根据计算每个输出特征块所需的输入特征图区域,将输入特征图在XY维度(也即Hi/Wi维度)上划分为Rs个输入特征块,以分配给Rs个从处理电路。可以理解,取决于卷积核尺寸和卷积步长,输出特征图上相邻的输出点所对应的输入特征图可能会存在重叠。
图8示出了根据本披露实施例的输入特征图的示例性拆分示意图。在此示例中,将输入特征图划分成16份分配在16个SL上,对应Group1模式。
图中810代表单个Co的输出特征图,其在XY方向上按照4×4方式划分成16个形状相同的输出特征块,分别分配给SL0~SL15。继而,由这16个输出特征块可以映射到输入特征图820上,获得分别计算这16个输出特征块所需的16个输入特征图区域,其同样是将输入特征图在XY方向上划分。这16个输入特征图区域可以相应地分配给16个从处理电路SL。
根据前文描述,会按照确定的卷积拆分方案,以拆分单元为单位对输入特征图进行拆分,因此,上述实施例中对输入特征图的分块要使得划分的每个输入特征图块在XY方向上是拆分单元XY方向维度的倍数,也即在XY方向上可以按照拆分单元对齐。例如,在选择4×4×4的卷积拆分方案时,每个输入特征图块按4×4对齐;而在选择16×2×2的卷积拆分方案时,每个输入特征图块按2×2对齐。
对于输出特征图不按拆分单元(例如4×4或2×2)对齐的情况,需要相应的在输入特征图上填补(例如补0),使得实际计算的输出XY是按拆分单元(例如4×4或2×2)对齐的并且输入XY也是按拆分单元(例如4×4或2×2)对齐的。
本领域技术人员可以理解,也可以在输出特征图的XY方向按照其他规则进行拆分,例如按照1×16方式拆分成16个形状相同的输出特征块,分别分配给SL0~SL15。本披露实施例在此方面没有限制。此外,还可以理解,虽然前面结合从处理电路之间的拆分进行描述,但是这种拆分 方式也可以应用于其他场景下的拆分,例如单个从处理电路SL内的运算电路CU之间的拆分,本披露实施例在此方面没有限制。
第二存储电路上的数据存储示例
如前所述,输入特征图或卷积核之一可以存储在图5的第一存储电路530上,二者中另一可以存储在第二存储电路540上。第一存储电路中的数据可以经由广播通路进行多播,而第二存储电路中的数据通常进行分发。通过合理地分配各个数据的存储方式,可以加快数据访存速度。在一些实施例中,第二存储电路可以为每个从处理电路SL分配一个存储区域,从而每个从处理电路运算所需的数据只需要从其对应的存储区域读取即可。
图9a-9d示出了根据本披露实施例的第二存储电路中的数据存储示意图。图中示例性示出了为例如Ns=16个从处理电路SL0~SL15分配的16块存储区域900~915。每个存储区域中存储该从处理电路要处理的卷积核或输入特征图。可以理解,取决于不同的分组模式,各个存储区域中的存储内容也会有所不用。
图9a示出了Group1模式下,输入特征图被拆分成16份FB0~FB15,存储在第二存储电路的各个存储区域中。对应每个SL的存储区域中存储一个连续的二维区域,这些二维区域是按照例如图8的方式拆分的。在每个二维区域中,按照前文描述的拆分单元按行存储,也即一行对应一个输入特征图的拆分单元。例如,假设拆分后的每个输入特征块包括4个拆分单元,也即4行数据,则在分配给SL0的存储区域1100中,按顺序存储有第一行Line01、第二行Line02、第三行Line03和第四行Line04输入特征图。每一行也可以称为一个输入特征行。
图9b示出了Group16模式下,卷积核根据Co划分,存储在第二存储电路的各个存储区域中,以分配给对应的SL。对应每个SL的存储区域中存储分配给其不同Co值的卷积核。例如,前文描述了两种Co分配方式,对应地,其存储方式也有两种。图9b中示出了其中一种,也即在每一轮运算中,将连续的Co值按顺序分配给各个SL。这样每轮运算完成后,各个SL输出的运算结果的Co维度是连续的。例如,图中示出第一轮运算中的Co=0~15的卷积核依次存储在16个存储区域900~915上;第二轮运算中的Co=16~31的卷积核顺次依次存储在16个存储区域900~915上;以此类推。可以理解,Group16模式下,也可能将输入特征图存储在第二存储电路上(未图示)。此时输入特征图无需拆分,直接被复制16份,分别存储在第二存储电路的各个存储区域中,以分配给对应的SL,从而每个SL可以针对相同的输入特征图、不同Co值的卷积核执行卷积运算。
图9c示出了Group4模式下一种可能的存储内容。在此图示示例中,输入特征图被拆分成4份并复制4份,存储在第二存储电路的各个存储区域中。具体地,每个从处理电路组SLB针对相同的输入特征图、不同Co的卷积核进行处理;而每个SLB内的4个SL分别处理一块拆分的输入特征图块。因此,图中用于4个SLB的存储区域的存储内容是相同的,例如900~903中的内容与912~915中的内容相同。进一步地,在每个SLB内,用于不同SL的存储区域存储不同的拆分输入特征块,例如900中存储输入特征块FB0,901中存储输入特征块FB1,以此类推。其他SLB的存储区域内也进行相同的存储分配,不再赘述。
图9d示出了Group4模式下另一种可能的存储内容。在此图示示例中,卷积核按照Co分成4组,存储在第二存储电路的各个存储区域中。具体地,卷积核按照Co以间隔1为单位划分至各组。例如,当Co=16时,通过多轮顺次分配给4个SLB,其中Co=0分配给G0{SL0~SL3},Co=1分配给的G1{SL4~SL7},Co=2分配给的G2{SL8~SL11},Co=3分配给G3{SL12~SL15};接着从Co=4开始又顺次分配给4个SLB。每个SLB内的4个SL共用相同权值。例如,存储区域900、901、902和903中存储相同的权值。同样地,Co也可以采用单个SLB内连续的方式,本领域技术人员参考前文描述可以推导其存储方式,此处不再详述。
单个从处理电路内的示例性卷积运算过程
在拆分好待运算数据并进行相应的摆放存储之后,就可以调度多个从处理电路对输入特征图 和卷积核的对应数据行执行卷积运算,继而可以根据卷积拆分方案,对多个从处理电路返回的运算结果进行拼接处理,以得到输入特征图和卷积核的卷积运算的输出特征图。具体地,可以利用从处理电路中的多个运算电路CU以及各个缓冲电路(参见图5)来执行具体的卷积运算过程。取决于从处理电路内部缓冲电路的空间大小以及运算电路的算力限制,在每轮运算中通常需要执行多次计算来完成所需运算。
在一些实施例中,第一缓冲电路可以用于缓存输入特征图,其可能来自第一存储电路或第二存储电路;相应地,第二缓冲电路可以用于缓存卷积核,其可能来自第二存储电路或第一存储电路。如前文所提到,以拆分单元(一行数据)为单位进行卷积运算处理,可以充分发挥硬件的算力,避免或减少无效计算。因此,每个运算电路CU可以在每次计算时,针对分别从第一缓冲电路中选取的数据行(例如输入特征行)和从第二缓冲电路中选取的数据行(例如,权值行)执行对位乘累加运算。为了简便起见,以下描述针对单个从处理电路SL内的处理,可以理解,其他SL内进行类似的处理。
从前面描述可知,在针对常规3D卷积运算场景下,单个从处理电路内的所有运算电路计算对应同一输出通道Co的一个输出特征图或部分输出特征图。取决于从处理电路SL内第一缓冲电路和第二缓冲电路的缓冲空间大小、运算电路CU的处理能力(例如内部寄存器等),从处理电路可能无法一次计算完分配给其的输出特征图。因此,可以以运算电路单次运算能力(例如,单次计算Nop个输出点或部分和)为单位,划分输出特征块,每个输出特征块对应单个SL内所有可调度的N
CU个运算电路的单次运算能力(N
CU*Nop个输出点)。例如,以前文图5中每个SL包括4个CU为例,假设每个CU单次可以计算Nop=4个输出点或输出点的部分和,则单个SL单次可以计算4*4=16个输出点(或部分和)。因此,可以将输出特征图在XoYo维度上按照16个输出点对齐划分输出特征块,逐个计算各个输出特征块。可以理解,这16个输出点可以按照4*4形式,也可以按照1*16形式,本披露实施例在此方面没有限制。
在计算每个划分的输出特征块时,又可以进一步在这N
CU个运算电路之间划分该输出特征块的输出点,以确定各个运算电路的处理对象。继而,可以根据输出点的划分,以拆分单元为滑动窗口,从第一缓冲电路中选取N
CU个输入特征数据行分发给N
CU个运算电路,从第二缓冲电路中选取对应的权值数据,广播给N
CU个运算电路,从而通过复用权值数据来实现多个滑动窗口对应的输出点的并行计算。执行Nk次滑动选取,其中Nk根据卷积核在X和Y维度的尺寸和从处理电路单次运算所支持的最大卷积核尺寸中的较小值来确定。
在一些实施例中,当执行三维卷积运算时,可以按如下选取对应的权值数据:从第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,将其复制Nop-1份扩展为一个扩展权值行,广播给从处理电路内的N
CU个运算电路。
此时,每个运算电路可以在每次滑动选数期间,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值数据行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和;以及将Nk个滑动选数期间计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到并输出Nop个运算结果。
从处理电路在输出其内运算电路的输出点时,可以根据输出点的划分方式,按特定顺序输出其内多个运算电路计算的输出点,以使得连续输出的输出点在X和/或Y维度上连续,方便后续处理。在一些实施例中,前面提到的分块电路可以进一步将从各个从处理电路返回的运算结果以第四维度存储顺序存储。根据情况,分块电路还可以将运算结果转换为期望的维度存储顺序存储。
运算电路之间输出点的划分可以有多种方式,相应地滑动选数卷积过程以及输出点的输出顺序也有所不同。
图10a-10b示出了运算电路之间两种不同的输出点划分示意图。
图10a示出了根据本披露一些实施例为每个运算电路分配连续输出点的示意图。在这些实施例中,可以在N
CU个运算电路之间将该输出特征块平均划分为N
CU个形状相同的输出特征子块,每个输出特征子块包括Nop个输出点,从而每个运算电路分别负责计算其中一个输出特征子块。例如,图中以上面示例为例,示出了输出特征块1010a包括4*4个输出点,平均划分的每个输出 特征子块1011a~1011d均包括2*2个输出点,每个运算电路每次计算连续的2*2个输出点(或部分和)。图中用不同的背景示出分配给4个不同运算电路CU0~CU3的输出点。
基于上述输出点划分,当通过滑动选数执行卷积运算时,可以根据计算输出特征子块所需的数据,与这N
CU个输出特征子块位置相对应地,从第一缓冲电路中选取N
CU个数据行进行运算。
例如,在输入特征数据的首次选数时,可以根据计算4个输出特征子块1011a~1014a所需的4个输入特征块,从对应的输入特征块中分别选取首个输入数据行,分发给4个运算电路。
在进行权值数据选取时,可以从第二缓冲电路中选取对应的权值数据,广播给N
CU个运算电路,从而通过复用权值数据来实现多个运算电路对应的输出点的并行计算。
进一步地,在一些实施例中,为了充分发挥运算装置CU内部的算力(例如乘加运算器),例如单次计算Nop个输出点或部分和,可以在单个输入数据行内进行权值复用,从而同时计算Nop个输出点或部分和。
例如,在权值数据的选数时,可以只取1/Nop个权值行,将其复制Nop-1份以扩展成1个权值行,此扩展权值行中包括Nop个相同的1/Nop权值行。扩展权值行同样可以广播给N
CU个运算电路,从而在多个运算电路之间复用权值的同时,在单个运算电路的Nop个输出点的计算之间以更小的粒度(例如1/Nop行)复用权值。
由此,通过每次对应地取N
CU个输入特征数据行、取1/Nop个权值行复制扩展成1个权值行,每次可以计算N
CU*Nop个输出点或部分和。当计算结果是部分和时,通过多次滑动,可以多次计算部分和,各次的部分和根据所属的输出点进行累加,可以得到最终结果。
根据输出点的划分方式,可以确定卷积运算的滑动次数和滑动步长。按照图10a的划分方式,滑动次数Nk=Kx*Ky,其中Kx、Ky分别是卷积核在X和Y维度的尺寸和从处理电路单次运算所支持的最大卷积核尺寸中的较小值,滑动步长=1。从处理电路单次运算所支持的最大卷积核尺寸例如由第一缓冲电路和第二缓冲电路的空间大小决定。可以理解,当卷积核超过最大卷积核尺寸时,需要在Kx和Ky方向按照该最大卷积核尺寸进行拆分。
按照图10a的划分方式,由于每个运算电路所计算的输出点在X和/或Y维度上是连续的,因此可以逐个输出各个运算电路的运算结果。例如按照运算电路的顺序,每次输出一个运算电路的运算结果,例如2*2个输出点,连续4次返回4*4的输出特征块。
图10b示出了根据本披露另一些实施例为每个运算电路分配间隔输出点的示意图。在这些实施例中,可以在N
CU个运算电路之间将该输出特征块平均划分为Nop个形状相同的输出特征子块,每个输出特征子块包括N
CU个输出点,分别划分给N
CU个运算电路。例如,图中仍以上面示例为例,示出了输出特征块1010b包括4*4个输出点,平均划分的每个输出特征子块1011b~1011b均包括2*2个输出点。在每个输出特征子块中,这2*2个输出点分配给4个运算电路。由此,每个运算电路计算Nop个输出特征子块中各一个输出点。图中用不同的背景示出分配给4个不同运算电路CU0~CU3的输出点。
基于上述输出点划分,当通过滑动选数执行卷积运算时,可以根据计算输出特征子块所需的数据,从每个输出特征子块的输出点位置相对应地,从第一缓冲电路中选取N
CU个数据行进行运算。
例如,在输入特征数据的首次选数时,可以根据计算首个输出特征子块1011b内的4个输出点所需的4个输入特征块,从对应的输入特征块中选取4个输入数据行,分发给4个运算电路。可以理解,由于这4个输出点在X和/或Y方向是连续的,因此同时选取的4个输入数据行在X和/或Y方向的间隔或步长是1。
在进行权值数据选取时,可以从第二缓冲电路中选取对应的权值数据,广播给N
CU个运算电路,从而通过复用权值数据来实现多个运算电路对应的输出点的并行计算。
进一步地,在一些实施例中,为了充分发挥运算装置CU内部的算力(例如乘加运算器),例如单次计算Nop个输出点或部分和,可以在单个输入数据行内进行权值复用,从而同时计算Nop个输出点或部分和。
例如,在权值数据的选数时,可以只取1/Nop个权值行,将其复制Nop-1份以扩展成1个权 值行,此扩展权值行中包括Nop个相同的1/Nop权值行。扩展权值行同样可以广播给N
CU个运算电路,从而在多个运算电路之间复用权值的同时,在单个运算电路的Nop个输出点的计算之间以更小的粒度(例如1/Nop行)复用权值。
由此,通过每次对应地取N
CU个输入特征数据行、取1/Nop个权值行复制扩展成1个权值行,每次可以计算N
CU*Nop个输出点或部分和。当计算结果是部分和时,通过多次滑动,可以多次计算部分和,各次的部分和根据所属的输出点进行累加,可以得到最终结果。
根据输出点的划分方式,可以确定卷积运算的滑动次数和滑动步长。按照图10b的划分方式,滑动次数Nk=ceil(Kx/2)*ceil(Ky/2),其中Kx、Ky分别是卷积核在X和Y维度的尺寸和从处理电路单次运算所支持的最大卷积核尺寸中的较小值,滑动步长=2。同样,从处理电路单次运算所支持的最大卷积核尺寸例如由第一缓冲电路和第二缓冲电路的空间大小决定。可以理解,当卷积核超过最大卷积核尺寸时,需要在Kx和Ky方向按照该最大卷积核尺寸进行拆分。
按照图10b的划分方式,由于每个运算电路所计算的输出点在X和/或Y维度上是间隔的,也即不连续的,因此每次需要选取部分运算电路的部分运算结果进行输出,以使得输出点在X和/或Y维度上连续。例如可以每次输出一行1*4个运算结果,连续4次返回4*4的输出特征块。在此示例中,第一行需要输出CU0的两个结果和CU1的两个结果,第二行则需要输出CU2的两个结果和CU3的两个结果,以此类推。在另一示例中,仍然可以每次输出2*2个运算结果,连续4次返回4*4的输出特征块。在此示例中,第一次输出CU0~CU3每个的首个运算结果,第二次输出CU0~CU3每个的第二个运算结果,依次类推。在另一示例中,还可以按列输出运算结果,此处不再赘述。
此外,考虑到利用运算电路CU内部的寄存器,一个从处理电路在Xo/Yo方向可以计算多个4*4区域,例如最多计算16个4*4的区域。此时,可以根据第二存储电路中的存储内容,可以复用权值或神经元,减少第二存储电路的读数频率。计算出的结果如果是部分和,则存储在运算电路内的寄存器上。
在这些实施例中,每个从处理电路可以根据权值复用和/或输入特征图复用方式,控制权值数据行和输入特征图数据行的读取方式,以通过多次运算将权值数据和输入特征图数据同时遍历卷积输出点的整个卷积窗口执行对位乘累加运算,得到多个部分和结果并累加得到对应卷积输出点上的卷积输出。
以下结合几个具体实施例描述采用不同卷积拆分方案、应用于不同类型的卷积运算时的详细运算过程。
实施例1:Forward16
在Forward16中,拆分单元的形状是16B×2×2,其运算过程也可以应用于与其类似的卷积拆分方案中。这些卷积拆分方案所指示的拆分单元的尺寸可以表示为Uci×Uy×Ux=M,Uci为拆分单元在输入特征图和卷积核初始的最低存储维度(例如Ci维度)上的尺寸,Ux和Uy分别为拆分单元在输入特征图和卷积核初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量。在这些卷积拆分方案中,Uci>Ux=Uy>1,Uci=M/4
n,
例如,假设M=64,则M/4
n可以是64、16、4和1,按照Uci>Ux=Uy>1这一规则,拆分单元可以是16B×2×2的形状。在使用这一卷积拆分方案时,需要将输入特征图和卷积核的Ci维度对齐到16B。例如,当Ci=40时,可以通过补零对齐到3*16=48,从而按照16B×2×2进行拆分,Ci维度上有3个拆分单元。
又例如,假设M=128,则M/4
n可以是128、32、8和2,按照Uci>Ux=Uy>1这一规则,拆分单元可以是32B×2×2、8B×4×4的形状。在使用这一卷积拆分方案时,需要将输入特征图和卷积核的Ci维度对齐到32B或8B。例如,当Ci=40时,可以通过补零对齐到2*32=64,也可以对齐到5*8=40,此时无需补零,因此可以优先按照8B×4×4进行拆分,Ci维度上有5个拆分单元。
因此,尽管下面结合Forward16的具体示例来描述卷积运算过程,但是这些运算过程也可以应用于这些与Forward16类似的卷积拆分方案中。
图11示出了根据本披露实施例1的Forward16方案的拆分和存储示意图。为了简便起见,图中示例假设数据类型为Int8。
图中1110示出了原始待运算数据(其可以是神经元或权值),其存储顺序是HWC。图中还示出了原始待运算数据按拆分单元进行拆分的4个数据块1111-1114,每个数据块包括16×2×2=64个数据。
图中1120示出了拆分后的数据摆放格式,以方便读取。可以看出,原始的数据块(例如1111-1114)被摆放成C维度上的一行(例如1121-1124)。在每行内,数据按照CHW的顺序存储,例如对于数据行1121,先存储C=0的4个数据,接着存储C=1的4个,然后是C=2的4个,一直到最后是C=15的4个。
具体而言,对于神经元来说,需要将数据从[1 Hi Wi Ci]摆放为:
[1*Hi/2*Wi/2*Ci/16*(16×2×2)],这种七维张量的形状。
对于权值而言,需要将数据从[Co Kh Kw Ci]摆放为:
[Co*Kh/2*Kw/2*Ci/16*(16×2×2)],这种七维张量的形状。
当使用图5所示的计算装置执行Forward16卷积拆分方案时,可以按照Forward16卷积拆分方案,由主处理电路内集成的分块电路、或完全或部分独立于主处理电路的分块电路将输入特征图和卷积核拆分成多个对应的拆分单元。分块电路还可以转换输入特征图和卷积核的维度存储顺序,以使得每个拆分单元内的数据连续存储为一个数据行。拆分和转换后的输入特征图和/或卷积核可被提供给主处理电路或从处理电路。继而,主处理电路可以将其获得的数据分发给多个从处理电路以供执行卷积运算;以及根据卷积拆分方案,对多个从处理电路返回的运算结果进行拼接处理,以得到输入特征图和卷积核的卷积运算的输出特征图。多个从处理电路则可以根据其获得的数据执行卷积运算,并向主处理电路返回运算结果。
在Forward16这种场景中,Co通常对齐到16。在这些实施例中,卷积拆分方案还可以指示执行卷积运算所需的运算轮次L,其中每个运算轮次中处理的输出通道Co数量对应于该运算轮次中可调度的从处理电路数量Ns,从而一个从处理电路可以处理一个Co值。
由于每个从处理电路处理不同的Co值,因此输入特征图可以在这些从处理电路之间复用。鉴于此,在一些实施例中,可以将输入特征图确定为多播数据,并将拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线传输给所调度的多个从处理电路。对应地,可以将卷积核确定为分发数据,并将拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以便分发给对应的从处理电路。这些分发数据可以在运算前分发给对应的从处理电路。
在此示例中,可以进一步将各个运算轮次中分配给各个从处理电路的、不同Co值的卷积核分别存储在第二存储电路中为对应的从处理电路分配的存储区域中。第二存储电路中的存储内容例如参考图9b。
相应地,第一缓冲电路可以缓存来自第一存储电路的、广播传输的多个输入特征数据行;而第二缓冲电路可以缓存来自第二存储电路的、分发给该从处理电路的卷积核的多个权值数据行。取决于具体的拆分和/或复用方式,这些数据行可以在运算期间被分发给对应的运算电路或广播给该从处理电路内的所有运算电路。继而,每个运算电路CU可以在每次运算中,针对分别从第一缓冲电路中选取的输入特征数据行和从第二缓冲电路中选取的权值数据行执行对位乘累加运算。
当单个从处理电路SL内的多个运算电路CU共同处理一个Co值时,需要在这多个CU之间对输出点进行拆分。在Forward16方案中,4个运算电路CU之间的输出点拆分方式例如可以参考图10a,也即,在每次计算时,每个运算电路计算输出特征图上在X和/或Y维度连续的多个输出点。
图12示出了根据本披露一个实施例的Forward16方案中的单次运算过程示意图。在该示例中,第一缓冲电路1210的大小为3×3×64B,也即最多可以缓存9行数据,第二缓冲电路1220的大小为2×2×64B,也即最多可以缓存4行数据。为了与拆分单元一致,图中的缓冲电路内的存储同样以拆分单元为单位示出。
图中示出了第一次滑动取数的运算过程。按照与输出点的划分方式对应的方式,以拆分单元为滑动窗口,从第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给N
CU个运算电路以供计算;从第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,其中Nop是每个运算电路单次最大可计算卷积输出点数量,将其复制Nop-1份扩展为一个扩展权值行,广播给从处理电路内的N
CU个运算电路。
具体地,在图5所示的计算装置中,N
CU=4,Nop=4。在划分输出点时,按照每次计算时,每个运算电路计算包括2×2个输出点的输出特征区块进行划分。
如图所示,从第一缓冲电路1210中在起始位置从与划分的输出点对应的4个输入特征块中各选取一个输入特征数据行对应地发送给该从处理电路SL内的4个运算电路1240。从第二缓冲电路1220中在起始位置选取1/4个权值数据行,将其复制3份扩展为一个扩展权值数据行1230,广播给该SL内的4个运算电路1240。
在每次计算时,每个运算电路针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和。
如图所示,4个运算电路1240对分发的输入特征数据行和广播的扩展权值数据行执行对位乘累加运算,得到运算结果1250。1250中不同背景颜色的结果代表由不同运算电路1240得到的。可以看出,每次计算时,一个CU会计算1个2×2的部分和,4个CU总计获得4个2×2的部分和,也即4×4。
接着,在第一缓冲电路和第二缓冲电路中同步滑动取数,进行下一计算。执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是卷积核在X和Y维度的尺寸或从处理电路在当前卷积拆分模式(也即Forward16)下单次运算所支持的最大卷积核尺寸中的较小值。相应地,运算电路将Nk次滑动计算期间计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到并输出Nop个运算结果。
在一些实施例中,在Forward16模式下,从处理电路单次运算所支持的最大卷积核尺寸为3×3。
图13示出了根据本披露一个实施例的Forward16方案中的滑动卷积过程示意图。该示例以6×6的输入特征图,3×3的卷积核为例,卷积步长为1,则输出特征图大小为4×4。输入特征图已对齐到2×2,分成9块16×2×2(C×H×W)大小的块,存储在第一缓冲电路中,图中示出为1310,其中省去了C维度。卷积核3×3则需要对齐到4×4,对齐部分补0,存储在第二缓冲电路中,图中示出为1320,同样省去了C维度。每次计算时,选取卷积核中1×1大小的块,复制3次,刚好对应上输入特征图的2×2的块,复制操作可以由硬件实现。
每次滑动时的输入特征图和卷积核在第一缓冲电路和第二缓冲电路中的选取范围如图13所示,共9幅图,代表共滑动9次。图中方块1310代表第一缓冲电路中的输入特征图,四个虚线框表示选择发给四个CU的区域;方块1320代表第二缓冲电路中的卷积核,虚线框代表选出的1/4行,其被复制3份扩展成一行后广播给4个CU。滑动次数Nk=Kx*Ky=9。
在每次计算时,每个CU针对来自第一缓冲电路的一个数据行和来自第二缓冲电路的一个扩展数据行,以1/4个数据行为单位进行对位乘累加,得到4个部分和;以及在当前运算轮次中将Nk次计算得到的对应同一卷积输出点的Nk个部分和进行累加,得到并输出4个运算结果。
具体地,对于图13中的每幅图,CU的个数Ncu=4,每个CU计算输出特征图上的4个输出点的部分和,该部分和是1/4个数据行的对位乘累加结果,也即每个输出点为一个16×1×1(Ci×Y×X)的标准卷积。滑动Nk=Kx*Ky=9次之后,Y×X方向完成累加,最终1个SL中得到完整的4×4(Y×X)的输出(如图10a所示)。对于更大的卷积核,需要在Kx和Ky方向按照上面同样的原理进行拆分运算。
图14示出了根据本披露一个实施例的Forward16方案中滑动卷积结果的累加示意图。
如图中1410所示,每个运算电路CU在每次计算时,针对来自第一缓冲电路的一个输入特征数据行和来自第二缓冲电路的一个扩展权值数据行,以1/4个数据行为单位进行对位乘累加,得到4个部分和。
每个运算电路CU在当前运算轮次中将Nk=Kx*Ky次计算得到的对应同一卷积输出点的Nk个部分和进行累加,得到4个运算结果。
可以理解,当Ci>16时,需要在Ci方向遍历,同时切换输入和权值,直到计算出完整的输出。当每个CU计算的Xo/Yo大于4时,需要沿着Xo/Yo方向滑动,读取不同的输入神经元和权值。本领域技术人员根据前述描述可以类似地推导出其计算过程,此处不再赘述。
从前面的滑动卷积过程可以看出,滑动模式输出的结果并不是传统卷积输出数据的正常排列顺序。因此,在输出过程中,各个从处理电路SL可以将其内运算电路CU的运算结果转换为指定的格式,例如Nco×Uy×Ux的格式。在一些实施例中,每个从处理电路可以按照输出点连续划分的顺序,每次输出其内一个运算电路的Nop个运算结果。分块电路可以进一步将从各个从处理电路返回的运算结果以第四维度存储顺序存储。根据情况,分块电路还可以将运算结果转换为期望的维度存储顺序存储。
图15示出了根据本披露一个实施例的Forward16拆分方案的输出数据格式示意图。
图中1510示出了1个SL的原始输出。从图中可以看出,每个CU计算2×2的输出神经元。由于1个CU计算的4个输出神经元是相邻的,因此每个SL可以按照输出点连续划分的顺序,每次可以输出其中一个CU的计算结果,也即1×2×2(Co×Y×X)的区域,连续4次返回1×4×4(Co×Y×X)的区域,也即4个CU中各个的4个运算结果。同一SL内的不同CU输出同一Co的输出特征图的不同区域。不同的SL输出不同Co的输出特征图。
图中1520示出了16个SL的存出数据结构。如图所示,输出缓存电路(例如图5的第三缓冲电路)可以将输出结果转换为16×2×2的格式,其中16对应SL的数量,也对应输出通道Co的数量。
在一些实施例中,考虑到运算电路内部寄存器的存储空间,例如包含四个运算电路的单个从处理电路最多可以计算16个4×4的输出特征区域,因此可以复用权值,从而减少第二存储电路的读数频率。也即,第一存储电路和第二存储电路的读数频率可以不同。运算电路计算出的结果如果是部分和,则存储在内的寄存器上。
在这些实施例中,从处理电路可以进一步用于:根据运算电路内的存储空间限制,确定从处理电路内的权值复用次数rs;以及控制第一缓冲电路中的输入特征数据的加载频次,以使得第二缓冲电路中每次加载的权值数据重复使用rs次,与第一缓冲电路中rs次加载的对应输入特征数据执行卷积运算。在一些示例中,rs可以取不大于16的值。
实施例2:Forward4
在Forward4中,拆分单元的形状是4B×4×4,其运算过程也可以应用于与其类似的卷积拆分方案中。这些卷积拆分方案所指示的拆分单元的尺寸可以表示为Uci×Uy×Ux=M,Uci为拆分单元在输入特征图和卷积核初始的最低存储维度(例如Ci维度)上的尺寸,Ux和Uy分别为拆分单元在输入特征图和卷积核初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量。在这些卷积拆分方案中,Ux=Uy≥Uci>1,Uci=M/4
n,
例如,假设M=64,则M/4
n可以是64、16、4和1,按照Ux=Uy≥Uci>1这一规则,拆分单元可以是4B×4×4的形状。在使用这一卷积拆分方案时,需要将输入特征图和卷积核的Ci维度对齐到4B。例如,当Ci=10时,可以通过补零对齐到3*4=12,从而按照4B×4×4进行拆分,Ci维度上有3个拆分单元。
又例如,假设M=128,则M/4
n可以是128、32、8和2,按照Ux=Uy≥Uci>1这一规则,拆分单元可以是2B×8×8的形状。在使用这一卷积拆分方案时,需要将输入特征图和卷积核的Ci维度对齐到2B。例如,当Ci=3时,可以通过补零对齐到2*2=4,从而按照2B×8×8进行拆分,Ci维度上有2个拆分单元。
因此,尽管下面结合Forward4的具体示例来描述卷积运算过程,但是这些运算过程也可以应用于这些与Forward4类似的卷积拆分方案中。
图16示出了根据本披露一个实施例的Forward4方案的拆分和存储示意图。为了简便起见, 图中示例假设数据类型为Int8。
图中1610示出了原始待运算数据(其可以是神经元或权值),其存储顺序是HWC。图中还示出了原始待运算数据按拆分单元进行拆分的4个数据块1611-1614,每个数据块包括4×4×4=64个数据。
图中1620示出了拆分后的数据摆放格式,以方便读取。可以看出,原始的数据块(例如1611-1614)被摆放成C维度上的一行(例如1621-1624)。在每行内,数据按照CHW的顺序存储,例如对于数据行1621,先存储C=0的16个数据,接着存储C=1的16个,然后是C=2的16个,最后是C=3的16个。
具体而言,对于神经元来说,需要将数据从[1 Hi Wi Ci]摆放为:
[1*Hi/4*Wi/4*Ci/4*(4×4×4)],这种七维张量的形状。
对于权值而言,需要将数据从[Co Kh Kw Ci]摆放为:
[Co*Kh/4*Kw/4*Ci/4*(4×4×4)],这种七维张量的形状。
当使用图5所示的计算装置执行Forward4卷积拆分方案时,可以按照Forward4卷积拆分方案,由主处理电路内集成的分块电路、或完全或部分独立于主处理电路的分块电路将输入特征图和卷积核拆分成多个对应的拆分单元。分块电路还可以转换输入特征图和卷积核的维度存储顺序,以使得每个拆分单元内的数据连续存储为一个数据行。拆分和转换后的输入特征图和/或卷积核可被提供给主处理电路或从处理电路。继而,主处理电路可以将其获得的数据分发给多个从处理电路以供执行卷积运算;以及根据卷积拆分方案,对多个从处理电路返回的运算结果进行拼接处理,以得到输入特征图和卷积核的卷积运算的输出特征图。多个从处理电路则可以根据其获得的数据执行卷积运算,并向主处理电路返回运算结果。
在Forward4这种场景中,输入通道数量较小的情况下,卷积核普遍也较小,例如Kh和Kw通常是个位数,Co与Ci的尺寸差不多。在这些实施例中,通常单轮运算中卷积核的输出通道Co维度的尺寸不超过所调度的从处理电路的数量,因此单个Co的运算需要由一个或多个从处理电路来完成。更一般地,即使Co维度较大时,也可以通过拆分成多轮运算来实现,其中每轮运算处理的Co尺寸不超过所调度的从处理电路的数量。由此,在一个示例中,可以首先基于卷积核的输出通道Co维度尺寸和可调度的从处理电路数量Ns,确定完成卷积运算所需的运算轮次以及各轮次运算中处理的Co数量或相应的分组模式。
Forward4卷积拆分方案可以支持前文结合图7描述的三种分组模式:Group1、Group4和Group16。
为了同时支持这三种分组模式,在一些实施例中,可以将卷积核确定为多播数据,并将拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线传输给所调度的多个从处理电路。对应地,可以将输入特征图确定为分发数据,并将拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以便分发给对应的从处理电路。这些分发数据可以在运算前分发给对应的从处理电路。输入特征图例如可以参考图8的示意在单个SLB的多个SL之间拆分。第二存储电路中的存储内容例如可以参考图9a(Group1)、图9c(Group4),未示出Group16模式下的存储内容。
相应地,第一缓冲电路可以缓存来自第二存储电路的、分发给该从处理电路的多个输入特征数据行;而第二缓冲电路可以缓存来自第一存储电路的、多播传输给该从处理电路的对应输出通道值的卷积核的多个权值数据行。取决于具体的拆分和/或复用方式,这些数据行可以在运算期间被分发给对应的运算电路CU或广播给该从处理电路内的所有CU。继而,每个运算电路CU可以在每次运算中,针对分别从第一缓冲电路中选取的输入特征数据行和从第二缓冲电路中选取的权值数据行执行对位乘累加运算。
当单个从处理电路SL内的多个运算电路CU共同处理一个Co值时,需要在这多个CU之间对输出点进行拆分。在Forward4方案中,4个运算电路CU之间的输出点拆分方式例如可以参考图10b,也即,在每次计算时,每个运算电路计算输出特征图上在X和/或Y维度间隔的多个输出点。
图17示出了根据本披露一个实施例的Forward4方案中的单次运算过程示意图。在该示例中,第一缓冲电路1710的大小为3×3×64B,也即最多可以缓存9行数据,第二缓冲电路1720的大小为2×2×64B,也即最多可以缓存4行数据。为了与拆分单元一致,图中的缓冲电路内的存储同样以拆分单元为单位示出。
图中示出了第一次滑动取数的运算过程。按照与输出点的划分方式对应的方式,以拆分单元为滑动窗口,从第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给N
CU个运算电路以供计算;从第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,其中Nop是每个运算电路单次最大可计算卷积输出点数量,将其复制Nop-1份扩展为一个扩展权值行,广播给从处理电路内的N
CU个运算电路。
具体地,在图5所示的计算装置中,N
CU=4,Nop=4。在划分输出点时,按照单次计算时,每个运算电路计算X和Y维度上均间隔1的2×2个输出点进行划分。
如图所示,从第一缓冲电路1710中在起始位置以及X和/或Y方向各移动1的位置选取一个输入特征数据行,总计选取4个输入特征数据行,对应地发送给该从处理电路SL内的4个运算电路1740。从第二缓冲电路1720中在起始位置选取1/4个权值数据行,也即选取2×2大小的数据,将其复制3份扩展为一个扩展权值数据行1730,广播给该SL内的4个运算电路1740。
在每次计算时,每个运算电路针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和。
如图所示,4个运算电路1740对分发的输入特征数据行和广播的扩展权值数据行执行对位乘累加运算,得到运算结果1750。1750中不同背景颜色的结果代表由不同运算电路1740得到的。可以看出,每次运算,一个CU会计算4个输出点的部分和,4个CU总计获得4×4的部分和。可以看出,每个CU计算的输出点在输出特征图的XoYo维度上并没有相邻。
接着,在第一缓冲电路和第二缓冲电路中同步滑动取数,进行下一计算。执行Nk次滑动选数,其中Nk=ceil(Kx/2)*ceil(Ky/2),Kx和Ky分别是卷积核在X和Y维度的尺寸或从处理电路在当前卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值。相应地,运算电路将Nk次滑动计算期间计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到Nop个运算结果。
在一些实施例中,在Forward4模式下,从处理电路单次运算所支持的最大卷积核尺寸为8×8。
图18示出了根据本披露一个实施例的Forward4方案中的滑动卷积过程示意图。该示例以9×9的输入特征图,5×5的卷积核为例,卷积步长为1,则输出特征图大小为5×5。输入特征图需要对齐到12×12,分成9块4×4×4(C×H×W)大小的块,存储在第一缓冲电路中,图中示出为1810,其中省去了C维度。卷积核5×5则需要对齐到8×8,对齐部分补0,存储在第二缓冲电路中,图中示出为1820,同样省去了C维度。每次计算时,选取卷积核中2×2大小的块,复制4次,刚好对应上输入特征图的4×4的块,复制操作可以由硬件实现。
每次滑动时的输入特征图和卷积核在第一缓冲电路和第二缓冲电路中的选取范围如图18所示,共9幅图,代表共滑动9次。图中方块1810代表第一缓冲电路中的输入特征图,四个虚线框表示选择发给四个CU的区域;方块1820代表第二缓冲电路中的卷积核,虚线框代表选出的1/4行,其被复制3份扩展成一行后广播给4个CU。滑动次数Nk=ceil(Kx/2)*ceil(Ky/2)=9。
在每次计算时,每个CU针对来自第一缓冲电路的一个输入特征数据行和来自第二缓冲电路的一个扩展权值数据行,以1/4个数据行为单位进行对位乘累加,得到4个部分和;以及在当前运算轮次中将Nk个运算周期内得到的对应同一卷积输出点的Nk个部分和进行累加,得到并输出4个运算结果。
具体地,对于图18中的每幅图,CU的个数Ncu=4,每个CU单次计算Nop=4个输出点或部分和,该部分和是1/4个数据行的对位乘累加结果,也即每个输出点为一个4×2×2(Ci×Y×X)的标准卷积。滑动Nk=ceil(Kx/2)*ceil(Ky/2)=9次之后,Y×X方向完成累加,最终1个SL中得到完整的4×4(Y×X)的输出(如图10b所示)。这种模式下单次计算仅支持卷积核不大于8×8 的情形,对于更大的卷积核,需要在Kx和Ky方向按照8×8进行拆分,可以按照上面同样的原理进行拆分运算。
可以理解,当Ci>4时,需要在Ci方向遍历,同时切换输入和权值,直到计算出完整的输出。当每个CU计算的Xo/Yo大于4时,需要沿着Xo/Yo方向滑动,读取不同的输入神经元和权值。本领域技术人员根据前述描述可以类似地推导出其计算过程,此处不再赘述。
从前面的滑动卷积过程可以看出,滑动模式输出的结果并不是传统卷积输出数据的正常排列顺序。因此,在输出过程中,各个从处理电路SL可以将其内运算电路CU的运算结果转换为指定的格式,例如Nco×Uy×Ux的格式。在一些实施例中,每个从处理电路可以每次输出其内部分运算电路的部分运算结果,该部分运算结果在输出特征图的X和/或Y维度上连续。分块电路可以进一步将从各个从处理电路返回的运算结果以第四维度存储顺序存储。根据情况,分块电路还可以将运算结果转换为期望的维度存储顺序存储。
当分组模式和/或单个SLB内输入特征图的拆分方式(也即根据输出特征图的HoWo拆分方式)不同时,输出的数据格式略有不同。
图19示出了根据本披露一个实施例Forward4方案的输出数据格式示意图。在此实施例中,分组模式为Group1,单个SLB(包括16个SL)内输入特征图的拆分方式按照Ho×Wo=1×16拆分。
图中1910示出了1个SL的原始输出。从图中可以看出,每个SL每次输出1×1×4(Co×Y×X)的区域,也即每次输出其内部分运算电路的部分运算结果,例如2个CU中各2个运算结果(参见图10b),这一部分运算结果在输出特征图的X和/或Y维度上连续,例如为同一行(图19所示)或同一列。连续4次返回1×4×4(Co×Y×X)的区域,也即4个CU中各个的4个运算结果。不同的SL输出同一Co的输出特征图的不同区域。当输出所有Co的4×4区域之后,继续输出会切换不同的输出点。
图中1920示出了16个SL的存出数据结构。如图所示,最终输出数据在写入存储电路(例如第一存储电路)后变为Yo*Xo*Co*4*16*4的格式,其中的Yo和Xo为每一个SL划分到的输出特征图的块的个数,16为在16个SL上的划分。根据需要,在一些实现中,可以再次进行摆数操作以转化为其他期望的数据格式。
如前面所提到,分组模式和/或单个SLB内多个SL之间输入特征图的拆分方式不同时,输出的数据格式还有细微的差别。假设原始输出大小为:
1*ho*wo*co
那么,Group1在Ho*Wo按照4*4拆分时的输出数据形状为:
ho/(4*4)*wo/(4*4)*co/group*(4*16*4)
上式中,(4*16*4)是forward4的基本输出块,方向分别对应h*c*w,其中16表示16个SL上的相同co的ho、wo的划分;ho,wo除了两次4,其中,第一个4表示在SL存储数据时进行4×4拆分,第二个4表示h、w方向的数据块折叠。Group1模式下,上面的group=1。
Group1在Ho*Wo按照1*16拆分时的输出数据形状为:
ho/(4)*wo/(4*16)*co/group*(4*16*4)
上式中,(4*16*4)是forward4的基本输出块,方向分别对应h*c*w,其中16表示16个SL上的相同co的ho、wo的划分;Group1模式下,上面的group=1。该形状也即图19示意图的形状。
由此可见,Group1的情况下,16个SL平分一个输出特征图的Yo*Xo维度。输出时行内维度SL中的数据,与16个SL在Yo*Xo方向上平分输出神经元的方式一一对应。此场景适合输入神经元Y*X方向数值大,Co数值小。
Group4输出数据形状为:
ho/(2*4)*wo/(2*4)*co/group*(4*16*4)
上式中,(4*16*4)含义同上,不同的是16表示4个co在4个SL上的wo输出划分。Group4模式下,上面的group=4。
Group16输出数据形状为:
ho/4*wo/4*co/group*(4*16*4)
上述中,(4*16*4)含义同上,不同的是16表示16个co在16个SL上的输出划分。Group16模式下,上面的group=16。
由于Group在H*W方向上还有不同的拆分类别,上述中4*16*4的16在具体的拆分上还有差异。由于Forwrd4是按照4B*4*4的块为计算单元,那么不可避免在计算时就存在对齐限制。根据不同的Group模式,相同Group模式的不同H*W的拆分方式,最终在计算时的对齐限制也不一样。在对齐的计算上,可以首先根据输出特征图的拆分方式确定ho*wo的对齐限制,再由ho*wo反推回hi*wi,由于输入神经元需要摆成拆分单元块的形式,从而还需要再对齐一次。上述对齐限制可以汇总如下表2:
表2、对齐限制
综上,在输出时,硬件可以自动按照行内4*16*4(Y*SL*X)维度,行间Y*X*C维度的方式输出神经元。对于更大的卷积核同理。
在一些实施例中,考虑到运算电路内部寄存器的存储空间,例如包括四个运算电路的单个从处理电路最多可以计算16个4×4的输出特征区域,因此可以复用输入特征图/神经元,从而减少第二存储电路的读数频率。也即,第一存储电路和第二存储电路的读数频率可以不同。运算电路计算出的结果如果是部分和,则存储在内的寄存器上。
在这些实施例中,从处理电路可以进一步用于:根据运算电路内的存储空间限制,确定从处理电路内的输入特征复用次数rn;以及控制第二缓冲电路中的权值数据的加载频次,以使得第一缓冲电路中每次加载的输入特征数据重复使用rn次,与第二缓冲电路中rn次加载的对应权值数据执行卷积运算。在一些示例中,rn可以取不大于16的值。
实施例3:Forward1
在Forward1中,拆分单元的形状与Forward4相同,也是4B×4×4;区别在于Forward1应用于深度卷积运算,也即2D卷积运算。2D卷积运算的原理可以参考前面结合图4b的描述。下面的描述可以应用于与Forward1类似的卷积拆分方案中。
由于在深度卷积中输入通道不进行累加,因此卷积核和输入特征图的维度都可以简化成C(通道)、H(高度)、W(宽度)三个维度。这些卷积拆分方案所指示的拆分单元的形状同样满足:Uc×Uy×Ux=M,Uc为拆分单元在输入特征图和卷积核初始的最低存储维度(例如C维度)上的尺寸,Ux和Uy分别为拆分单元在输入特征图和卷积核初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量。在这些卷积拆分方案中,Ux=Uy≥Uc>1,Uc=M/4
n,
Forward1方案中数据的拆分和存储示例可以参考实施例2中对Forward4的描述,例如参考图16,相同部分不再重复。对于Forward1方案而言,只需将Ci、Co维度替换成C维度,也即将输入特征图和卷积核都简化为三维数据。
具体而言,对于神经元来说,需要将数据从[Hi Wi C]摆放为:
[Hi/4*Wi/4*C/4*(4×4×4)],这种六维张量的形状,省略N维度。
对于权值而言,需要将数据从[Kh Kw C]摆放为:
[Kh/4*Kw/4*C/4*(4×4×4)],这种六维张量的形状,省略N维度。
当使用图5所示的计算装置执行Forward1卷积拆分方案时,可以按照Forward1卷积拆分方案,由主处理电路内集成的分块电路、或完全或部分独立于主处理电路的分块电路将输入特征图和卷积核拆分成多个对应的拆分单元。分块电路还可以转换输入特征图和卷积核的维度存储顺序,以使得每个拆分单元内的数据连续存储为一个数据行。拆分和转换后的输入特征图和/或卷积核可被提供给主处理电路或从处理电路。继而,主处理电路可以将其获得的数据分发给多个从处理电路以供执行卷积运算;以及根据卷积拆分方案,对所调度的多个从处理电路返回的运算结果进行拼接处理,以得到输入特征图和卷积核的卷积运算的输出特征图。多个从处理电路则可以根据其获得的数据执行卷积运算,并向主处理电路返回运算结果。
在Forward1这种深度卷积运算场景中,由于C维度上的运算结果不需要累加,因此不同C上的运算分配在不同的运算电路上可以相对独立地进行。需要注意,在采用Forward1拆分方案中,C维度会按4B对齐,因此,当以拆分单元为单位进行处理时,C维度会对齐到4B(也即Uc)再进行拆分。换言之,不同运算电路上的处理在C维度上是以Uc为单位进行拆分的。
在深度卷积场景中,通常通道数C较小,而卷积核和输入特征图普遍较大。在这些实施例中,通常单轮运算中输入特征图和卷积核的通道C维度Nc对Uc的倍数不超过所调度的从处理电路的数量,因此以Uc为单位计算的单个通道的运算可以由一个或多个从处理电路来完成。更一般地,即使C维度较大时,也可以通过拆分成多轮运算来实现,其中每轮运算处理的C维度尺寸Nc对Uc的倍数不超过所调度的从处理电路的数量。由此,在一个示例中,可以首先基于卷积核的通道C维度尺寸和可调度的从处理电路数量Ns,确定完成卷积运算所需的运算轮次以及各轮次运算中处理的C数量Nc或相应的分组模式,其中Nc对齐到Uc。
类似于Forward4,Forward1方案也可以支持前文结合图7描述的三种分组模式:Group1、Group4和Group16。Forward1与Forward4的分组模式的区别在于,Forward1中C维度的拆分是以Uc为单位进行,例如每4个连续的C(对应一个Uc)分配给一个分组(或从处理电路组SLB)。由此,在一些实施例中,根据单轮运算中卷积核的C维度尺寸Nc和可调度的从处理电路数量Ns,将Nc按照Uc对齐,确定每Rs个从处理电路处理对应同一Uc的卷积核和输入特征图,Rs=[Ns/(Nc/Uc)],表示从处理电路之间的权值复用次数。
为了同时支持这三种分组模式,在一些实施例中,可以将卷积核确定为多播数据,并将拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线传输给所调度的多个从处理电路。对应地,可以将输入特征图确定为分发数据,并将拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以便分发给对应的从处理电路。这些分发数据可以在运算前分发给对应的从处理电路。
输入特征图例如可以参考图8的示意在单个SLB的多个SL之间拆分。具体地,可以按如下在每Rs个从处理电路之间划分对应Uc的输入特征图:根据输出特征图的尺寸,在XY维度上平均划分为Rs个形状相同的输出特征块;根据计算每个输出特征块所需的输入特征图区域,在XY维度上将输入特征图划分为Rs个输入特征块;以及将Rs个输入特征块分别按拆分单元进行拆分并转换维度存储顺序后存储在第二存储电路中为Rs个从处理电路分配的存储区域中。
第二存储电路中的存储内容例如可以参考图9a(Group1)、图9c(Group4),未示出Group16模式下的存储内容。
相应地,第一缓冲电路可以缓存来自第二存储电路的、分发给该从处理电路的多个输入特征数据行;而第二缓冲电路可以缓存来自第一存储电路的、多播传输给该从处理电路的对应输出通道值的卷积核的多个权值数据行。取决于具体的拆分和/或复用方式,这些数据行可以在运算期间被分发给对应的运算电路CU或广播给该从处理电路内的所有CU。继而,每个运算电路CU可以在每次运算中,针对分别从第一缓冲电路中选取的输入特征数据行和从第二缓冲电路中选取的权值数据行执行对位乘累加运算。
当单个从处理电路SL内的多个运算电路CU共同处理一个Uc时,需要在这多个CU之间对 输出点进行拆分。类似于Forward4中,在Forward1中也按照每个运算电路分配间隔输出点的方式进行划分(例如图10b)。但是在Forward1中,卷积核的划分更小,按照4×4为单位拆分,而Forward4中卷积核按照8×8为单位拆分,因此,输出点的划分略有不同。具体地,在一个实施例中,在每次计算时,每个运算电路计算输出特征图上在X和/或Y维度间隔的1个输出点;以及在不同次计算中,每个运算电路计算输出特征图上在X和/或Y维度上不同的输出点。
图20示出了根据本披露一个实施例的Forward1方案中运算电路的输出点划分示意图。
由于卷积核按照4×4为单位拆分,每次计算时只需要使用第二缓冲电路中的1行权值,而第一缓冲电路内可以存放的输入特征数据最多9行,因此最多可以计算8×8的输出。图中示出了4个运算电路在8×8的输出点上的划分,其中用不同的背景示出分配给4个不同运算电路CU0~CU3的输出点。由于每次计算时权值只有1行,因此每次计算即可得到当前的输出点,无需进行滑动累加。
例如,第一次滑动时,4个CU分别计算第一子块2001中的4个输出点;第二次向右滑动时,4个CU分别计算第二子块2002中的4个输出点,以此类推。8×8个输出点相应地需要滑动16次。
图21示出了根据本披露一个实施例的Forward1方案中的单次运算过程示意图。在该示例中,第一缓冲电路2110的大小为3×3×64B,也即最多可以缓存9行数据,第二缓冲电路2120的大小为2×2×64B,也即最多可以缓存4行数据。为了与拆分单元一致,图中的缓冲电路内的存储同样以拆分单元为单位示出。
图中示出了第一次滑动取数的运算过程。按照与输出点的划分方式对应的方式,以拆分单元为滑动窗口,从第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给从处理电路内的N
CU个运算电路以供计算;从第二缓冲电路中读取1个权值数据行,广播给从处理电路内的N
CU个运算电路。
具体地,在图5所示的计算装置中,N
CU=4,Nop=4。
如图所示,从第一缓冲电路2110中在起始位置以及X和/或Y方向各移动1的位置选取一个输入特征数据行,总计选取4个输入特征数据行,对应地发送给该从处理电路SL内的4个运算电路2140。从第二缓冲电路2120中在起始位置选取1个权值数据行,也即选取4×4大小的数据2130,将广播给该SL内的4个运算电路2140。
在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个权值行,以1/Uc个数据行为单位,将对应同一通道值的特征数据和权值数据进行对位乘累加,得到Uc个输出点。
如图所示,4个运算电路2140对分发的输入特征数据行和广播的权值数据行按照1/Uc(Uc=4)行执行对位乘累加运算,得到运算结果2150。2150中不同背景颜色的结果代表由不同运算电路2140得到的。可以看出,每次运算,一个CU计算Uc上各个XoYo面上的1个输出点,4个CU总计获得Uc×2×2的输出点。可以看出,4个CU计算的输出点在输出特征图的XoYo维度是相邻的。
接着,在第一缓冲电路滑动取数,第二缓冲电路中无需滑动,仍然使用这一行权值进行下一计算。在第一缓冲电路上执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是卷积核在X和Y维度的尺寸或从处理电路在当前卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值。相应地,运算电路将Nk次滑动计算期间计算得到的Nk*Uc个输出点按照输出点的划分方式进行拼接,得到Uc个通道上的Nk*N
CU个运算结果。
在一些实施例中,在Forward1模式下,从处理电路单次运算所支持的最大卷积核尺寸为4×4。
图22示出了根据本披露一个实施例的Forward1方案中的滑动卷积过程示意图。该示例以11×11的输入特征图,4×4的卷积核为例,卷积步长为1,则输出特征图大小为8×8。输入特征图需要对齐到12×12,分成9块4×4×4(C×H×W)大小的块,存储在第一缓冲电路中,图中示出为2210,其中省去了C维度。卷积核按照4×4拆分,存储在第二缓冲电路中,图中示出为 2220,同样省去了C维度。每次计算时,选取4×4大小的卷积核,刚好对应上输入特征图的4×4的块,广播给4个运算电路。
每次滑动时的输入特征图和卷积核在第一缓冲电路和第二缓冲电路中的选取范围如图22所示,共16幅图,代表共滑动16次。图中方块2210代表第一缓冲电路中的输入特征图,四个虚线框表示选择发给四个CU的区域;方块2220代表第二缓冲电路中的卷积核,虚线框代表选出的1个权值行,其被广播给4个CU,并且在滑动过程中无需重选。滑动次数Nk=Kx*Ky,其中Kx、Ky分别是卷积核在X和Y维度的尺寸和从处理电路单次运算所支持的最大卷积核尺寸中的较小值,滑动步长=2。同样,从处理电路单次运算所支持的最大卷积核尺寸例如至少由第一缓冲电路和第二缓冲电路的空间大小决定。可以理解,当卷积核超过最大卷积核尺寸时,需要在Kx和Ky方向按照该最大卷积核尺寸进行拆分。
在每次计算时,每个CU针对来自第一缓冲电路的一个输入特征数据行和来自第二缓冲电路的一个权值数据行,按照1/Uc行进行对位乘累加,得到Uc上各个XoYo面上的1个输出点,从而N
CU个运算电路每次得到Uc个XoYo面上的N
CU个输出点。可以理解,当滑动经过Nk个运算周期的运算之后,可以得到Uc个XoYo面上的Nk*N
CU个输出点,将其进行拼接,可以得到C维度上Uc个面上的最大8×8(Ho*Wo)个输出点,也即Uc×8×8。
具体地,对于图22中的每幅图,CU的个数Ncu=4,每个CU单次计算C维度上Uc个面上的1个输出点,该部分和是1/Uc(1/4)个数据行的对位乘累加结果,也即每个输出点为一个4×4(Y×X)的2D卷积。滑动Nk=Kx*Ky=16次之后,完成最大输出点计算,1个SL中得到8×8(Y×X)的输出(如图20所示)。这种模式下单次计算仅支持卷积核4×4的情形,对于更大的卷积核,需要在Kx和Ky方向按照4×4进行拆分,可以按照上面同样的原理进行拆分运算。
可以理解,当每个CU计算的Xo/Yo大于8时,需要沿着Xo/Yo方向滑动,读取不同的输入神经元和权值。本领域技术人员根据前述描述可以类似地推导出其计算过程,此处不再赘述。
从前面的滑动卷积过程可以看出,滑动模式输出的结果并不是传统卷积输出数据的正常排列顺序。因此,在输出过程中,各个从处理电路SL可以将其内运算电路CU的运算结果转换为指定的格式,例如Nc×Uy×Ux的格式。在一些实施例中,每个从处理电路可以每次输出其内部分运算电路的部分运算结果,这些部分运算结果在输出特征图的X和/或Y维度上连续。分块电路可以进一步将从各个从处理电路返回的运算结果以第四维度存储顺序存储。根据情况,分块电路还可以将运算结果转换为期望的维度存储顺序存储。
当分组模式和/或单个SLB内输入特征图的拆分方式(也即根据输出特征图的HoWo拆分方式)不同时,输出的数据格式略有不同。
图23示出了根据本披露一个实施例Forward1方案的输出数据格式示意图。在此实施例中,分组模式为Group1,单个SLB(包括16个SL)内输入特征图的拆分方式按照Ho×Wo=1×16拆分。
图中2310示出了1个SL的原始输出。从图中可以看出,每个SL每次输出Uc×1×8(C×Y×X)的区域,也即每次输出其内部分运算电路的部分运算结果,例如2个CU中各4个运算结果(参见图20),这一部分运算结果在输出特征图的X和/或Y维度上连续,例如为同一行(图20所示)或同一列。连续8次返回Uc×8×8(C×Y×X)的区域,也即4个CU中各个的16个运算结果。不同的SL输出同一Uc的输出特征图的不同区域。当输出Uc的所有8×8区域之后,继续输出会切换不同的输出点。
图中2320示出了16个SL的存出数据结构。如图所示,最终输出数据在写入存储电路(例如第一存储电路)后变为Yo*Xo*ceil[C/Uc]*Uc*8*16*8的格式,其中的Yo和Xo为每一个SL划分到的输出特征图的块的个数,16为在16个SL上的划分。根据需要,在一些实现中,可以再次进行摆数操作以转化为其他期望的数据格式。
如前面所提到,分组模式和/或单个SLB内多个SL之间输入特征图的拆分方式不同时,输出的数据格式还有细微的差别。假设原始输出大小为:
1*ho*wo*c
那么,Group1在Ho*Wo按照4*4拆分时的输出数据形状为:
ho/(4*4)*wo/(4*4)*c/group/Uc*Uc*(8*16*8)
上式中,(8*16*8)是forward1的基本输出块,方向分别对应h*c*w,其中16表示16个SL上的相同Uc的ho、wo的划分;ho,wo除了两次4,其中,第一个4表示在SL存储数据时进行4×4拆分,第二个4表示h、w方向的数据块折叠。Group1模式下,上面的group=1。
Group1在Ho*Wo按照1*16拆分时的输出数据形状为:
ho/(4)*wo/(4*16)*c/group/Uc*Uc*(8*16*8)
上式中,(8*16*8)是forward1的基本输出块,方向分别对应h*c*w,其中16表示16个SL上的相同Uc的ho、wo的划分;Group1模式下,上面的group=1。该形状也即图23示意图的形状。
由此可见,Group1的情况下,16个SL平分一个输出特征图的Yo*Xo维度。输出时行内维度SL中的数据,与16个SL在Yo*Xo方向上平分输出神经元的方式一一对应。此场景适合输入神经元Y*X方向数值大,c数值小。
Group4输出数据形状为:
ho/(2*4)*wo/(2*4)*c/group/Uc*Uc*(8*16*8)
上式中,(8*16*8)含义同上,不同的是16表示4个Uc在4个SL上的wo输出划分。Group4模式下,上面的group=4。
Group16输出数据形状为:
ho/4*wo/4*co/group/Uc*Uc*(8*16*8)
上述中,(8*16*8)含义同上,不同的是16表示16个Uc在16个SL上的输出划分。Group16模式下,上面的group=16。
由于Group在H*W方向上还有不同的拆分类别,上述中8*16*8的16在具体的拆分上还有差异。由于Forwrd1是按照4B*4*4的块为计算单元,那么不可避免在计算时就存在对齐限制。根据不同的Group模式,相同Group模式的不同H*W的拆分方式,最终在计算时的对齐限制也不一样。在对齐的计算上,可以首先根据输出特征图的拆分方式确定ho*wo的对齐限制,再由ho*wo反推回hi*wi,由于输入神经元需要摆成拆分单元块的形式,从而还需要再对齐一次。上述对齐限制可以汇总如下表3:
表3、Forward1对齐限制
综上,在输出时,硬件可以自动按照行内8*16*8(Y*SL*X)维度,行间Y*X*C维度的方式输出神经元。对于更大的卷积核同理。
在一些实施例中,考虑到运算电路内部寄存器的存储空间,例如包括四个运算电路的单个从处理电路最多可以计算16个4×4的输出特征区域,因此可以复用输入特征图/神经元,从而减少第二存储电路的读数频率。也即,第一存储电路和第二存储电路的读数频率可以不同。运算电路计算出的结果如果是部分和,则存储在内的寄存器上。
在这些实施例中,从处理电路可以进一步用于:根据运算电路内的存储空间限制,确定从处理电路内的输入特征复用次数rn;以及控制第二缓冲电路中的权值数据的加载频次,以使得第一缓冲电路中每次加载的输入特征数据重复使用rn次,与第二缓冲电路中rn次加载的对应权值数 据执行卷积运算。在一些示例中,rn可以取不大于16的值。
实施例4:Update1
在Update1中,拆分单元的形状与Forward1相同,也是4B×4×4;区别在于Update1应用于神经网络模型的反向训练中的深度卷积运算,具体用于深度卷积运算中的反向训练中的权值更新过程,而Forward1应用于前向深度卷积运算,二者都是2D卷积运算。反向深度卷积运算的原理可以参考前面结合图4b的描述。反向深度卷积运算场景中,top_diff与bottom_data的尺寸通常都比较大,因此需要不同的优化运算方案。
在下面的描述中,虽然会以top_diff和bottom_data来指代待运算的数据,但是前面针对卷积核的描述可以类似地应用于top_diff,而针对输入特征图的描述可以类似地应用于bottom_data,也即二者可互换使用。下面的描述可以应用于与Update1类似的卷积拆分方案中。
由于在深度卷积中输入通道不进行累加,因此top_diff和bottom_data的维度都可以简化成C(通道)、H(高度)、W(宽度)三个维度。这些卷积拆分方案所指示的拆分单元的形状同样满足:Uc×Uy×Ux=M,Uc为拆分单元在bottom_data和top_diff初始的最低存储维度(例如C维度)上的尺寸,Ux和Uy分别为拆分单元在bottom_data和top_diff初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量。在这些卷积拆分方案中,Ux=Uy≥Uc>1,Uc=M/4
n,
由于深度卷积运算中,C通道维度的乘积结果不进行累加,因此,原本运算器在执行常规3D卷积时,例如C维度上64个数乘64个数,累加后得到1个数,现在却会得到64个数。也即,由于C维度上不累加浪费了运算器的算力,给运算器带来了性能损失。为了充分利用运算器的算力,通过上述拆分方式,将需要累加的维度(例如HW维度)上的数据转移到C维度上,从而可以提高运算器的使用率。例如,在采用4B×4×4的拆分单元时,假设数据类型为int8,则64个数乘64个数的累加结果得到4个数,而不是原来的64个数。
Update1方案中数据的拆分和存储示例可以参考实施例2中对Forward4的描述,例如参考图16,相同部分不再重复。对于Update1方案而言,只需将Ci、Co维度替换成C维度,也即将输入特征图(bottom_aata)和卷积核(top_diff)都简化为三维数据。
具体而言,对于bottom_data来说,需要将数据从[HiWiC]摆放为:
[Hi/4*Wi/4*C/4*(4×4×4)],这种六维张量的形状,省略N维度。
对于top_diff而言,需要将数据从[HoWoC]摆放为:
[Ho/4*Wo/4*C/4*(4×4×4)],这种六维张量的形状,省略N维度。
当使用图5所示的计算装置执行Update1卷积拆分方案时,可以按照Update1卷积拆分方案,由主处理电路内集成的分块电路、或完全或部分独立于主处理电路的分块电路将bottom_data和top_diff拆分成多个拆分单元。分块电路还可以转换bottom_data和top_diff的维度存储顺序,以使得每个拆分单元内的数据连续存储为一个数据行。拆分和转换后的bottom_data和/或top_diff可被提供给主处理电路或从处理电路。继而,主处理电路可以将其获得的数据分发给多个从处理电路以供执行卷积运算;以及根据卷积拆分方案,对所调度的多个从处理电路返回的运算结果进行拼接处理,以得到bottom_data和top_diff的深度卷积运算的输出△W(或称为weight_diff)。多个从处理电路则可以根据其获得的数据执行卷积运算,并向主处理电路返回运算结果。
为了充分利用可调度的从处理电路,可以在各个从处理电路之间分配相应的运算任务,以提高并行处理效率。考虑到Update1这种深度卷积运算场景中,C维度上的运算结果不需要累加,因此不同C上的运算分配在不同的运算电路上可以相对独立地进行。需要注意,在采用Update1拆分方案中,C维度会按4B对齐,因此,当以拆分单元为单位进行处理时,C维度会对齐到4B(也即Uc)再进行拆分。换言之,不同运算电路上的处理在C维度上是以Uc为单位进行拆分的。
在反向深度卷积场景中,C维度通常较大,例如大于64,bottom_data和top_diff普遍也较大。在这些实施例中,通常单轮运算中bottom_data和top_diff的通道C维度大小Nc可以是64的倍数,因此以Uc为单位计算的单个通道的运算可以分配在一个从处理电路来完成。由此,在一些 实施例中,卷积拆分方案还指示执行深度卷积运算的分组划分方式,其中分组划分方式可以针对bottom_data和top_diff数据,按照通道C维度、以Uc为单位顺次划分至可调度的Ns个从处理电路,每个从处理电路处理不同的连续Uc个C值的bottom_data和top_diff数据。换言之,单个从处理电路可以是一个组,分别处理不同C(以Uc为单位)的运算,也即对应前文的Group16分组模式。
在这种按照C维度划分分组的实施例中,可以将top_diff按照前述卷积拆分方案拆分并转换维度后存储在第一存储电路中。由于每个从处理电路处理不同的Uc,因此可以在运算期间通过广播总线将对应不同的Uc个C值的top_diff单播/分别传输给所调度的Ns个从处理电路。
在这些实施例中,可以将bottom_data确定为分发数据,并将拆分并转换维度存储顺序后的分发数据按照通道C维度、以Uc为单位顺次划分的方式分别存储在第二存储电路中与Ns个从处理电路对应的存储区域内,以便分发给对应的从处理电路。
图24示出了根据本披露一些实施例的bottom_data在第二存储电路中的示意性存储方式。
如图所示,第二存储电路可以为每个从处理电路分配一个存储区域,从而每个从处理电路运算所需的bottom_data数据只需要从其对应的存储区域读取即可。图中示例性示出了为16个从处理电路分配了16块存储区域2400~2415,每个存储区域中存储该从处理电路要处理的bottom_data数据块。
如前面所提到的,C维度上是以Uc为单位进行拆分的。在图中示例中,假设Uc=4B,数据类型为int8,则一个Uc包括4个C值。当C维度大小超过可调度的从处理电路数量的Uc倍时,需要通过多个运算轮次来执行运算。
以图中示例为例,假设总共16个从处理电路均可调度,进一步假设bottom_data数据的C维度大小为128,超过可调度的从处理电路数量的Uc倍(16*4=64),则可以分成两轮运算来完成全部计算。bottom_data按照C维度并以Uc为单位可以拆分成32个bottom_data数据块,前16个数据块在第一轮运算中计算,后16个数据块在第二轮运算中计算。
如图所示,在第一轮运算的数据中,包括C=0,1,2,3的bottom_data数据块分配给第一从处理电路;包括C=4,5,6,7的bottom_data数据块分配给第二从处理电路;以此类推。在第二轮运算的数据中,也类似地划分bottom_data数据块并相应地进行存储,此处不再重复。
相应地,第一缓冲电路可以缓存来自第二存储电路的、分发给该从处理电路的多个bottom_data数据行;而第二缓冲电路可以缓存来自第一存储电路的、单播传输给该从处理电路的对应Uc的多个top_diff数据行。取决于具体的拆分和/或复用方式,这些数据行可以在运算期间被分发给对应的运算电路CU或广播给该从处理电路内的所有CU。继而,每个运算电路CU可以在每次运算中,针对分别从第一缓冲电路中选取的bottom_data数据行和从第二缓冲电路中选取的top_diff数据行执行对位乘累加运算。
当单个从处理电路SL内的多个运算电路CU共同处理一个Uc时,需要在这多个CU之间对输出点进行拆分。类似于Forward4中,在Update1中也按照每个运算电路分配间隔输出点的方式进行划分(例如图10b)。在Update1中,卷积核top_diff按照4×4为单位拆分,而bottom_data数据每次只使用第一缓冲电路中的2×2个64B,因此,在第一缓冲电路上多次滑动取数计算后,最多可以计算4×4个输出点。
具体地,在一个实施例中,在每次计算时,每个运算电路计算输出△W的Uc个通道C值的XY面上、在X和/或Y维度相邻的1个输出点;以及在不同次计算中,每个运算电路计算输出△W上在X和/或Y维度上不同的输出点。滑动次数Nk=ceil(Kx/2)*ceil(Ky/2),其中Kx和Ky分别是输出△W在X和Y维度的尺寸或从处理电路在当前卷积拆分模式下单次运算所支持的最大输出尺寸中的较小值。例如,对于Kx=Ky=4的情况,Nk=2*2=4次,以即滑动4次,每次计算2×2个输出点,总共计算4×4个输出点。
Update1方案中的单次运算过程可以与Forward1类似,可以参考结合图21的描述,此处不再重复。
图25示出了根据本披露一个实施例的Update1方案中的滑动卷积过程示意图。在该示例中, 第一缓冲电路缓存2*2=4个bottom_data数据行,图中示出为2510,其中省去了C维度;第二缓冲电路则缓存1个top_diff数据行,图中示出为2520,同样省去了C维度。每个数据行都是4×4×4(C×H×W)大小的块。△W在X和Y维度的尺寸Kx=Ky=4。每次计算时,从第二缓冲电路选取4×4大小的top_diff,刚好对应上bottom_data的4×4的块,广播给4个运算电路。
具体地,按照与输出点的划分方式对应的方式,以拆分单元为滑动窗口,从第一缓冲电路中滑动选取N
CU个bottom_data数据行,分别发送给从处理电路内的N
CU个运算电路以供计算。进一步的,从第二缓冲电路中读取1个top_diff数据行,广播给从处理电路内的N
CU个运算电路。在第一缓冲电路上执行Nk次滑动选数,其中Nk=ceil(Kx/2)*ceil(Ky/2),Kx和Ky分别是权值梯度数据△W在X和Y维度的尺寸或从处理电路在当前卷积拆分模式下单次运算所支持的最大输出尺寸中的较小值。
每次滑动时的bottom_data和top_diff在第一缓冲电路和第二缓冲电路中的选取范围如图25所示,共4幅图,代表共滑动4次。图中方块2510代表第一缓冲电路中的bottom_data,四个虚线框表示选择发给四个CU的区域;方块2520代表第二缓冲电路中的top_diff,虚线框代表选出的1个top_diff数据行,其被广播给4个CU,并且在滑动过程中无需重选。滑动次数Nk=4,滑动步长=2。在Update1卷积运算模式下,从处理电路单次运算所支持的最大△W尺寸为4×4。可以理解,当△W超过最大支持尺寸时,需要在XY方向按照该最大支持尺寸进行拆分。
在每次计算时,每个CU针对来自第一缓冲电路的一个bottom_data数据行和来自第二缓冲电路的一个top_diff数据行,以1/Uc个数据行为单位,将对应同一通道值的bottom_data和top_diff数据进行对位乘累加,得到Uc个输出点,也即△W在Uc上各个KxKy面上的1个输出点,从而N
CU个运算电路每次得到Uc个KxKy面上的N
CU个输出点。可以理解,当滑动经过Nk个运算周期的运算之后,每个运算电路计算得到Uc个KxKy面上X和/或Y维度上间隔的Nk个输出点。N
CU个运算电路的Nk次滑动总计可以得到Uc个KxKy面上的Nk*N
CU个输出点。这些输出点进行拼接,可以构成C维度上Uc个面上的最大4×4(Kx*Ky)个输出点,也即Uc×4×4。
具体地,对于图25中的每幅图,CU的个数Ncu=4,每个CU单次计算C维度上Uc个面上的1个输出点,该部分和是1/Uc(1/4)个数据行的对位乘累加结果,也即每个输出点为一个4×4(Y×X)的2D卷积。滑动Nk=4次之后,完成最大输出点计算,1个SL中得到4×4(Y×X)的输出(如图10b所示)。
可以理解,当每个CU计算的Kx/Ky大于4时,需要沿着Kx/Ky方向滑动,读取不同的bottom_data和top_diff。本领域技术人员根据前述描述可以类似地推导出其计算过程,此处不再赘述。
从前面的滑动卷积过程可以看出,滑动模式输出的结果并不是传统卷积输出数据的正常排列顺序。因此,在输出过程中,各个从处理电路SL可以将其内运算电路CU的运算结果转换为指定的格式。在一些实施例中,每个从处理电路可以每次输出其内一个运算电路运算得到的Uc个XY面上同一位置的1个输出点。Ns个从处理电路每次同时输出Ns*Uc个XY面上同一位置的1个输出点。通过这种输出方式,该Ns*Uc个输出点在C维度上是连续的。分块电路可以进一步将从各个从处理电路返回的运算结果以第四维度存储顺序存储,例如按照Ky*Kx*(Ns*Uc)维度顺序进行拼接存储。根据情况,分块电路还可以将运算结果转换为期望的维度存储顺序存储。
图26示出了根据本披露一个实施例Update1方案的输出数据格式示意图。在此实施例中,采用按照C维度划分分组的方式,也即每个从处理电路SL处理不同Uc的运算。
图中2610示出了1个SL的原始输出。从图中可以看出,每个SL每次输出Uc×1×1(C×Y×X)的区域,也即每次输出其内一个运算电路的Uc个运算结果,例如CU0的4个运算结果,这4个运算结果在输出数据的C维度上连续。由于不同SL处理不同Uc的运算,因此16个SL可以同时输出不同Uc上、XY面上同一位置的1输出点,其可以在C维度上拼接成16*Uc个输出点,其在C维度上连续。
图中2620示出了16个SL的存出数据结构。如图所示,每次16个SL的输出拼接成C维度上连续的一行数据。例如,第一次,16个SL都输出Ky=0,Kx=0位置(标记“1”)的输出点; 第二次,16个SL都输出Ky=0,Kx=1位置(标记“2”)的输出点;以此类推。最终输出数据在写入存储电路(例如第一存储电路)后变为Kh*Kw*(16*Uc)的格式,其中16为在16个SL上的划分。根据需要,在一些实现中,可以再次进行摆数操作以转化为其他期望的数据格式。
在一些实施例中,考虑到运算电路内部寄存器的存储空间,例如包括四个运算电路的单个从处理电路最多可以计算16个4×4的输出点区域,因此可以复用bottom_data,从而减少第二存储电路的读数频率。也即,第一存储电路和第二存储电路的读数频率可以不同。运算电路计算出的结果如果是部分和,则存储在内部寄存器上。
在这些实施例中,从处理电路可以进一步用于:根据运算电路内的存储空间限制,确定从处理电路内的bottom_data复用次数rn;以及控制第二缓冲电路中的top_diff数据的加载频次,以使得第一缓冲电路中每次加载的bottom_data数据重复使用rn次,与第二缓冲电路中rn次加载的对应tap_diff数据执行卷积运算。在一些示例中,rn可以取不大于16的值。
实施例5:Update4
在Update4中,拆分单元的形状与Update1相同,也是4B×4×4;区别在于Update4应用于神经网络模型的反向训练中的叉乘卷积运算,具体用于叉乘卷积运算中的反向训练中的权值更新过程,而Update1应用于反向深度卷积运算。反向叉乘卷积运算的原理可以参考前面结合图4c的描述。由于反向叉乘卷积运算的特点,需要不同的优化运算方案。
在下面的描述中,虽然会以top_diff和bottom_data来指代待运算的数据,但是前面针对卷积核的描述可以类似地应用于top_diff,而针对输入特征图的描述可以类似地应用于bottom_data,也即二者可互换使用。下面的描述可以应用于与Update4类似的卷积拆分方案中。
这些卷积拆分方案所指示的拆分单元的形状同样满足:Uc×Uy×Ux=M,Uc为拆分单元在bottom_data和top_diff初始的最低存储维度(例如bottom_data为Ci维度,top_diff为Co维度)上的尺寸,Ux和Uy分别为拆分单元在bottom_data和top_diff初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量。在这些卷积拆分方案中,Ux=Uy≥Uc>1,Uc=M/4
n,
Update4方案中数据的拆分和存储示例可以参考实施例2中对Forward4的描述,例如参考图16,相同部分不再重复。
具体而言,对于bottom_data来说,需要将数据从[HiWiCi]摆放为:
[Hi/4*Wi/4*Ci/4*(4×4×4)],这种六维张量的形状,省略N维度。
对于top_diff而言,需要将数据从[HoWoCo]摆放为:
[Ho/4*Wo/4*Co/4*(4×4×4)],这种六维张量的形状,省略N维度。
当使用图5所示的计算装置执行Update4卷积拆分方案时,可以按照Update4卷积拆分方案,由主处理电路内集成的分块电路、或完全或部分独立于主处理电路的分块电路将bottom_data和top_diff拆分成多个对应的拆分单元。分块电路还可以转换bottom_data和top_diff的维度存储顺序,以使得每个拆分单元内的数据连续存储为一个数据行。拆分和转换后的bottom_data和/或top_diff可被提供给主处理电路或从处理电路。继而,主处理电路可以将其获得的数据分发给多个从处理电路以供执行卷积运算;以及根据卷积拆分方案,对所调度的多个从处理电路返回的运算结果进行拼接处理,以得到bottom_data和top_diff的深度卷积运算的输出△W(或称为weight_diff)。多个从处理电路则可以根据其获得的数据执行卷积运算,并向主处理电路返回运算结果。
从前文结合图4c描述的Update4所应用的交叉卷积运算原理可以看出,输出△W(权值梯度)包括四个维度[Co Kh Kw Ci],其中Co维度上的运算结果是相对独立的。因此,不同Co的运算分配在不同的运算电路上可以相对独立地进行。需要注意,在采用Update4拆分方案中,C维度会按4B对齐,因此,当以拆分单元为单位进行处理时,C维度会对齐到4B(也即Uc)再进行拆分。换言之,不同运算电路上的处理在Co维度上是以Uc为单位进行拆分的。
由此,在一些实施例中,可以首先基于输出通道Co维度尺寸和可调度的从处理电路数量Ns,确定完成该叉乘卷积运算的运算轮次、各轮次运算中处理的输出通道Co数量Nco以及相应的分 组模式,其中Nco对齐到Uc。
在一些实施例中,根据Co的不同取值范围,可以采用不同的分组模式来执行卷积运算。在一个实现中,当Co较小时,例如在1~4时,可以采用Group1分组模式,也即所有从处理电路SL属于一个组,共同处理同一Co(也即一个Uc)的运算。在另一实现中,当Co较大,例如在4~16时,可以采用Group4分组模式,也即所有SL分为4组,每组处理一个Co(也即一个Uc)的运算。在又一实现中,当Co非常大,例如超过16时,可以采用Group16分组模式,也即每个SL是一个组,每个SL处理不同Co(也即不同Uc)的运算。虽然上面示例性描述了不同Co范围适合的分组模式,但是也可以不按上述规则选择。例如,Co=16时,也可以采用Group1分组模式,通过多轮运算来完成所需处理。由此可见,不同组之间的拆分方式(例如,Group1、Group4、Group16)是根据Co来确定的。上述分组模式可以概括为GroupN,表示当前轮次运算中调度的Ns个从处理电路分为N组,每个从处理电路组处理相同的连续Uc个Co值,不同从处理电路组处理不同的连续Uc个Co值,N=4n,n=0,1,2…。
进一步地,在每个组内,也可以为对应数量的从处理电路分配运算任务,例如按照Ci维度。对于GroupN分组模式,假设每组有Rs个从处理电路,其中Rs=Ns/N,则需要在每个从处理电路组SLB内的Rs个从处理电路上分配运算任务,各组内分配方式相同。
当组内划分方式为按照Ci维度时,可以将bottom_data数据按照输入通道Ci方向,以Uc为单位顺次划分至同一组内的Rs个从处理电路上。top_diff数据无需进行额外处理。此时,可以将top_diff数据确定为多播数据,并将拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线将对应不同Uc个Co值的top_diff数据分别传输给所调度的N个从处理电路组,每个从处理电路组共享相同的Uc个Co值的神经元梯度数据。进一步地,可以将bottom_data数据确定为分发数据,并将拆分并转换维度存储顺序后的分发数据复制N份,每份按照Ci方向的分组划分方式划分为Rs个数据块,分别存储在第二存储电路中对应的存储区域内,以便分发给对应的从处理电路。
以下针对不同分组模式的划分进行详细阐述。
在Group1分组模式下,也即所有从处理电路SL共同处理同一Co的情况下,top_diff可以直接以拆分单元拆分后存储在第一存储电路中;bottom_data则除了按拆分单元进行拆分之外,还按照Ci维度、以Uc为单位划分成Ns个数据块存储在第二存储电路,以分发给Ns个从处理电路。
在这种按照Ci维度划分的实施例中,由于每个从处理电路处理不同Ci,因此可以广播同一Co(以Uc为单位)的top_diff给对应的从处理电路。进一步地,主处理电路可以将bottom_data确定为分发数据,并将拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以便分发给对应的从处理电路。
图27a示出了根据本披露一些实施例的Update4方案中Group1模式下第二存储电路中的示例性存储内容。
如图所示,bottom_data存储在第二存储电路中,其包括16个存储区域2700~2715,分别分配给16个从处理电路SL0~SL15。每个存储区域存储对应不同C(也即Ci)维度的bottom_data数据块。具体地,按照C维度间隔1个Uc,顺次分配给16个存储区域。例如,Ci=0~3分配给SL0,Ci=4~7分配给SL1,直至Ci=60~63分配给SL15;然后再从SL0开始分配。
在Group4分组模式下,每Rs=Ns/4个从处理电路构成一个从处理电路组SLB,共同处理同一Co。当按照Ci维度划分分组时,top_diff可以同样直接以拆分单元拆分后存储在第一存储电路中。由于每个SLB处理不同的Co,因此可以单播不同Co的top_diff给对应的SLB,SLB内的SL共享同一Co。换言之,同一Co的top_diff会多播给一个SLB内的多个SL。
在这些实施例中,每个SLB处理相同的bottom_data数据,在SLB内的Rs个SL之间,bottom_data数据按照Ci维度拆分成Rs=Ns/4份,存储在第二存储电路中对应的存储区域内,以便分发给对应的从处理电路。
图27b示出了根据本披露一些实施例的Update4方案中Group4模式下按C维度划分时第二 存储电路中的示例性存储内容。
如图所示,第二存储电路同样包括16个存储区域2700~2715,分别分配给16个从处理电路SL0~SL15。在Group4模式下,这16个存储区域根据对应的SLB也分成4个组,每组存储相同、完整的bottom_data,也即将bottom_data复制4份存储在4个SLB对应的存储区域中。
具体地,每个SLB针对相同的bottom_data、不同Co的top_diff进行处理;而每个SLB内的4个SL分别处理一块拆分的bottom_data数据块。这些bottom_data数据块根据Ci维度进行拆分,具体地,按照Ci维度间隔1个Uc,顺次分配给1个SLB内的4个SL的对应存储区域。因此,图中用于4个SLB的存储区域的存储内容是相同的,例如2700~2703中的内容与2712~2715中的内容相同。进一步地,在每个SLB内,用于不同SL的存储区域存储不同的拆分bottom_data数据块,例如2700中存储Ci=0~3的bottom_data数据块BD0,2701中存储Ci=4~7的bottom_data数据块BD1,以此类推。其他SLB的存储区域内也进行相同的存储分配,不再赘述。
在Group16分组模式下,也即每个从处理电路处理不同Co的情况下,与前文Update1方案类似,在16个SL之间可以按照Ci维度进行划分。此部分可以参考前文Update1方案的描述,不再赘述。
相应地,第一缓冲电路可以缓存来自第二存储电路的、分发给该从处理电路的多个bottom_data数据行;而第二缓冲电路可以缓存来自第一存储电路的、单播、多播或广播传输给该从处理电路的对应Co(以Uc为单位)的多个top_diff数据行。取决于具体的拆分和/或复用方式,这些数据行可以在运算期间被分发给对应的运算电路CU或广播给该从处理电路内的所有CU。继而,每个运算电路CU可以在每次运算中,针对分别从第一缓冲电路中选取的bottom_data数据行和从第二缓冲电路中选取的top_diff数据行执行对位乘累加运算。
当单个从处理电路SL内的多个运算电路CU共同处理以Uc为单位的一个Co时,需要在这多个CU之间对输出点进行拆分。由于Update4方案所应用的叉乘卷积运算的特性,CU之间可以按照输出通道Co维度来拆分输出点。例如,在Update4中,top_diff按照4×4为单位拆分,而bottom_data数据每次只使用第一缓冲电路中的2×2个64B,因此,在第一缓冲电路上多次滑动取数计算后,最多可以计算4×4×4×4(Co×Ky×Kx×Ci)个输出点。
具体地,在一个实施例中,在每次计算时,每个运算电路计算输出△W不同Co上、以Uc为单位的一个Ci上(也即连续Uc个Ci值上)、XY维度上同一位置的1个输出点。在不同次计算中,每个运算电路计算输出△W上在XY维度上不同的输出点。滑动次数Nk=Kx*Ky,其中Kx和Ky分别是输出△W在X和Y维度的尺寸或从处理电路在当前卷积拆分模式下单次运算所支持的最大输出尺寸中的较小值。例如,对于Kx=Ky=4的情况,Nk=4*4=16次,以即滑动16次,每次计算4×1×1×4(Co×Ky×Kx×Ci)个输出点,滑动16次总共计算4×4×4×4(Co×Ky×Kx×Ci)个输出点。
图28示出了根据本披露实施例的Update4方案中的单次运算过程示意图。在该示例中,第一缓冲电路2810的大小为3×3×64B,也即最多可以缓存9行数据,第二缓冲电路2820的大小为2×2×64B,也即最多可以缓存4行数据。为了与拆分单元一致,图中的缓冲电路内的存储同样以拆分单元为单位示出。
图中示出了第一次滑动取数的运算过程。按照与输出点的划分方式对应的方式,以拆分单元为滑动窗口,从第一缓冲电路中滑动选取1个bottom_data数据行,广播传输给从处理电路内的N
CU个运算电路以供计算;从第二缓冲电路中读取1个top_diff数据行,将Co(一个Uc)拆分成Uc个Co,每个Co的XY数据面复制Uc份,分别发送给从处理电路内的Uc个运算电路。在图中示例中,N
CU=4,数据类型为Int8,因此Uc=4。
如图所示,从第一缓冲电路2810中在起始位置选取一个bottom_data数据行,广播给该从处理电路SL内的4个运算电路2840。从第二缓冲电路2820中在起始位置选取1个top_diff数据行,也即选取4×4×4大小的数据2830,将其在Co维度上拆分成4个1×4×4的数据面,每个数据面复制4份,扩展成4×4×4(Ho×Wo×Ci)的数据行,分别发送给该SL内的4个运算电路2840。
在每次计算时,每个运算电路2840针对来自第一缓冲电路的一个bottom_data数据行和来自 第二缓冲电路的一个扩展top_diff数据行,以1/Uc=1/4个数据行为单位,将对应同一输入通道Ci值的bottom_data数据和top_diff数据进行对位乘累加,得到所分配Co值在Ci维度上的Uc个输出点。
如图所示,4个运算电路2840对广播的bottom_data数据行和分发的top_diff扩展数据行按照1/4行执行对位乘累加运算,得到运算结果2850。2850中不同背景颜色的结果代表由不同运算电路2840得到的。可以看出,每次运算,一个CU计算所分配的一个Co的Uc(Ci维度)上各个KxKy面上的1个输出点,4个CU总计获得4个Co上1×1×Uc的输出点。可以看出,4个CU计算的输出点对应不同Co的KxKy维度上的同一位置。
接着,在第一缓冲电路滑动取数,第二缓冲电路中无需滑动,仍然使用这一行top_diff数据进行下一计算。在第一缓冲电路上执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是△W在X和Y维度的尺寸或从处理电路在当前卷积拆分模式下单次运算所支持的最大输出尺寸中的较小值。相应地,每个运算电路在Nk次滑动计算期间计算得到Nk*Uc个输出点,其为单个Co上、Uc个Ci上的XY面上在X和/或Y维度上连续的Nk个输出点。4个运算电路则总共可以得到4个Co上Uc个Ci上的XY面上的Nk个运算结果。
在一些实施例中,在Update4模式下,从处理电路单次运算所支持的最大输出尺寸为4×4。
图29示出了根据本披露一个实施例的Update4方案中的滑动卷积过程示意图。在该示例中,第一缓冲电路缓存2*2=4个bottom_data数据行,图中示出为2910,其中省去了C维度;第二缓冲电路则缓存1个top_diff数据行,图中示出为2920,同样省去了C维度。每个数据行都是4×4×4(C×H×W)大小的块。△W在X和Y维度的尺寸Kx=Ky=4。每次计算时,从第二缓冲电路选取4×4大小的top_diff,并按C拆分复制,扩展成4个数据行,分发给4个运算电路。
每次滑动时的bottom_data和top_diff在第一缓冲电路和第二缓冲电路中的选取范围如图29所示,共16幅图,代表共滑动16次。图中方块2910代表第一缓冲电路中的bottom_data,虚线框表示选择广播给四个CU的区域;方块2920代表第二缓冲电路中的top_diff,虚线框代表选出的1个top_diff数据行,其被复制扩展后分发给4个CU,并且在滑动过程中无需重选。滑动次数Nk=16,滑动步长=1。在Update4卷积运算模式下,从处理电路单次运算所支持的最大△W尺寸为4×4。可以理解,当△W超过最大支持尺寸时,需要在XY方向按照该最大支持尺寸进行拆分。
在每次计算时,每个CU针对来自第一缓冲电路的一个bottom_data数据行和来自第二缓冲电路的一个top_diff扩展数据行,按照1/Uc行进行对位乘累加,得到△W在一个Co上、Uc个Ci上各个KxKy面上的1个输出点,从而N
CU个运算电路每次得到N
CU个Co上、Uc个KxKy面上的1个输出点。可以理解,当滑动经过Nk个运算周期的运算之后,可以得到N
CU个Co上、Uc个KxKy面上的Kx*Ky个输出点,将其进行拼接,可以得到N
CU个Co上、Uc个面上的最大4×4(Kx*Ky)个输出点,也即Ky×Kx×N
CU×Uc(Ky×Kx×Co×Ci)。
具体地,对于图29中的每幅图,CU的个数Ncu=4,每个CU单次计算一个Co上、Ci维度上Uc个面上的1个输出点,该输出点是1/Uc(1/4)个数据行的对位乘累加结果,也即每个输出点为一个4×4(Y×X)的2D卷积。滑动Nk=16次之后,完成最大输出点计算,1个SL中得到4×4×4×4(Y×X×Co×Ci)的输出。
可以理解,当每个CU计算的Kx/Ky大于4时,需要沿着Kx/Ky方向滑动,读取不同的bottom_data和top_diff。本领域技术人员根据前述描述可以类似地推导出其计算过程,此处不再赘述。
上文描述了单个从处理电路SL内的运算过程。当数据类型为Int16或Float16时,Uc=2,此时分配给每个从处理电路的Co只有2个,无法利用全部的运算电路CU。在这种情况下,可以额外再读一次数,将后两个Co的top_diff数据行读进来,由此仍然可以每个CU计算一个Co。当数据类型为Float32时,Uc=1,也即数据是在C维度是按照Co=1个数据拆分的。这种情况下,在一些实施例中,每次只计算一个Co,此时每个CU每拍只能计算一个Co,连续计算四拍,从而把4个Co都计算完。例如,CU0可以在第一拍计算Co=0的数据,CU1可以在第二拍计算Co=1的数据,以此类推。在另一些实施例中,也可以额外再读3次数,将后3个Co的top_diff数据行 读进来,由此仍然可以每个CU计算一个Co。也即,当Uc<N
CU时,可以从第二缓冲电路中读取N
CU/Uc个top_diff数据行,按照Co维度拆分成N
CU个Co值,每个Co值对应的XY数据面复制Uc份,分别发送给从处理电路内的N
CU个运算电路。
从前面的滑动卷积过程可以看出,滑动模式输出的结果并不是传统卷积输出数据的正常排列顺序。因此,在输出过程中,各个从处理电路SL可以将其内运算电路CU的运算结果转换为指定的格式。在一些实施例中,每个从处理电路可以每次输出其内一个运算电路的一次运算结果,这些运算结果是输出数据在一个Co上、Uc个Ci上的XY面上同一位置的1个输出点。也即,这些Uc个输出点在Ci维度上连续。同一SLB内的Rs个从处理电路每次同时输出同一Co的、Rs*Uc个Ci的XY面上同一位置的1个输出点。通过这种输出方式,该Rs*Uc个输出点在Ci维度上是连续的。分块电路可以进一步将从各个从处理电路返回的运算结果以第四维度存储顺序存储,例如按照Ky*Kx*Co/N*N*(Rs*Uc)维度顺序进行拼接存储,其中N代表GroupN,也即分组数。根据情况,分块电路还可以将运算结果转换为期望的维度存储顺序存储。
当分组模式(例如,Group1、Group4、Group16)不同时,输出的数据格式略有不同。
图30示出了根据本披露一个实施例Update4方案的输出数据格式示意图。在此实施例中,采用Group1分组模式、按照Ci维度划分分组的方式,也即每个从处理电路SL处理同一Co(以Uc为单位)、不同Ci(以Uc为单位)的输出数据的运算。
图中3010示出了1个SL的原始输出。从图中可以看出,每个SL每次输出1×Uc×1×1(Co×Ci×Y×X)的区域,也即每次输出其内一个运算电路的运算结果,例如CU0计算的Co=0上、Ci=0~3上4个XY面内Kx=Ky=0的1个输出点,这4个输出点在输出数据的Ci维度上连续,在XY维度上对应同一位置。由于16个SL处理相同Co、不同Ci的运算(均以Uc为单位),因此这16个SL可以同时输出相同Co、不同Ci上、XY面上同一位置的1个输出点,其可以在Ci维度上拼接成16*Uc个输出点,其在Ci维度上连续。
图中3020示出了16个SL的存出数据结构。如图所示,每次16个SL的输出拼接成Ci维度上连续的一行数据。例如,在第一个滑动计算周期后,16个SL可以首先输出其内CU0的运算结果,也即Co=0,Ky=0,Kx=0位置(标记“1”)的输出点;接着16个SL输出其内CU1的运算结果,也即Co=1,Ky=0,Kx=0位置(标记“1”)的输出点;直至都输出CU3的运算结果。在第二滑动计算周期后,16个SL可以依次输出对应Ky=0,Kx=1位置(标记“2”)的输出点;以此类推。最终输出数据在写入存储电路(例如第一存储电路)后变为Kh*Kw*Co*(16*Uc)的格式,其中16为在16个SL上的划分。根据需要,在一些实现中,可以再次进行摆数操作以转化为其他期望的数据格式。
如前面所提到,分组模式的拆分方式不同时,输出的数据格式还有细微的差别,其例如可以表示为Ky*Kx*Co/N*N*(Rs*Uc),其中N是GroupN中的分组数。
由于Update4是按照4B*4*4的块为计算单元,那么不可避免在计算时就存在对齐限制。根据不同分组模式(例如Group1、Group4、Group16等),最终在计算时的对齐限制也不一样。本领域技术人员可以根据不同数据类型、不同分组模式,推算出对各个数据的对齐限制,此处不再详述。
在一些实施例中,考虑到运算电路内部寄存器的存储空间,例如包括四个运算电路的单个从处理电路最多可以计算16个4×4的输出点区域,因此可以复用bottom_data,从而减少第二存储电路的读数频率。也即,第一存储电路和第二存储电路的读数频率可以不同。运算电路计算出的结果如果是部分和,则存储在内部寄存器上。
在这些实施例中,从处理电路可以进一步用于:根据运算电路内的存储空间限制,确定从处理电路内的bottom_data复用次数rn;以及控制第二缓冲电路中的top_diff数据的加载频次,以使得第一缓冲电路中每次加载的bottom_data数据重复使用rn次,与第二缓冲电路中rn次加载的对应top_diff数据执行卷积运算。在一些示例中,rn可以取不大于16的值。
以上结合Forward16、Forward4、Forward1、Update1和Update4的具体卷积拆分方案对本披露实施例提供的卷积优化方案进行了示例性描述和阐释。基于本披露的教导,本领域技术人员可 以根据具体的硬件电路配置(诸如从处理电路的个数、从处理电路内的运算电路的个数、硬件单次处理能力等)设想出其他的卷积拆分方案,其均落入本披露公开范围内,此处不再一一枚举。
本披露实施例还提供了利用前述计算装置执行卷积运算的方法。本领域技术人员可以理解,执行卷积运算的方法步骤与前面结合附图描述的计算装置的各个电路相对应,因此前面描述的特征同样适用于方法步骤,此处不再重复。
本披露实施例还提供了一种芯片,其可以包括前面结合附图描述的任一实施例的计算装置。进一步地,本披露还提供了一种板卡,该板卡可以包括前述芯片。
根据不同的应用场景,本披露的电子设备或装置可以包括服务器、云端服务器、服务器集群、数据处理装置、机器人、电脑、打印机、扫描仪、平板电脑、智能终端、PC设备、物联网终端、移动终端、手机、行车记录仪、导航仪、传感器、摄像头、相机、摄像机、投影仪、手表、耳机、移动存储、可穿戴设备、视觉终端、自动驾驶终端、交通工具、家用电器、和/或医疗设备。所述交通工具包括飞机、轮船和/或车辆;所述家用电器包括电视、空调、微波炉、冰箱、电饭煲、加湿器、洗衣机、电灯、燃气灶、油烟机;所述医疗设备包括核磁共振仪、B超仪和/或心电图仪。本披露的电子设备或装置还可以被应用于互联网、物联网、数据中心、能源、交通、公共管理、制造、教育、电网、电信、金融、零售、工地、医疗等领域。进一步,本披露的电子设备或装置还可以用于云端、边缘端、终端等与人工智能、大数据和/或云计算相关的应用场景中。在一个或多个实施例中,根据本披露方案的算力高的电子设备或装置可以应用于云端设备(例如云端服务器),而功耗小的电子设备或装置可以应用于终端设备和/或边缘端设备(例如智能手机或摄像头)。在一个或多个实施例中,云端设备的硬件信息和终端设备和/或边缘端设备的硬件信息相互兼容,从而可以根据终端设备和/或边缘端设备的硬件信息,从云端设备的硬件资源中匹配出合适的硬件资源来模拟终端设备和/或边缘端设备的硬件资源,以便完成端云一体或云边端一体的统一管理、调度和协同工作。
需要说明的是,为了简明的目的,本披露将一些方法及其实施例表述为一系列的动作及其组合,但是本领域技术人员可以理解本披露的方案并不受所描述的动作的顺序限制。因此,依据本披露的公开或教导,本领域技术人员可以理解其中的某些步骤可以采用其他顺序来执行或者同时执行。进一步,本领域技术人员可以理解本披露所描述的实施例可以视为可选实施例,即其中所涉及的动作或模块对于本披露某个或某些方案的实现并不一定是必需的。另外,根据方案的不同,本披露对一些实施例的描述也各有侧重。鉴于此,本领域技术人员可以理解本披露某个实施例中没有详述的部分,也可以参见其他实施例的相关描述。
在具体实现方面,基于本披露的公开和教导,本领域技术人员可以理解本披露所公开的若干实施例也可以通过本文未公开的其他方式来实现。例如,就前文所述的电子设备或装置实施例中的各个单元来说,本文在考虑了逻辑功能的基础上对其进行拆分,而实际实现时也可以有另外的拆分方式。又例如,可以将多个单元或组件结合或者集成到另一个***,或者对单元或组件中的一些特征或功能进行选择性地禁用。就不同单元或组件之间的连接关系而言,前文结合附图所讨论的连接可以是单元或组件之间的直接或间接耦合。在一些场景中,前述的直接或间接耦合涉及利用接口的通信连接,其中通信接口可以支持电性、光学、声学、磁性或其它形式的信号传输。
在本披露中,作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元示出的部件可以是或者也可以不是物理单元。前述部件或单元可以位于同一位置或者分布到多个网络单元上。另外,根据实际的需要,可以选择其中的部分或者全部单元来实现本披露实施例所述方案的目的。另外,在一些场景中,本披露实施例中的多个单元可以集成于一个单元中或者各个单元物理上单独存在。
在另外一些实现场景中,上述集成的单元也可以采用硬件的形式实现,即为具体的硬件电路,其可以包括数字电路和/或模拟电路等。电路的硬件结构的物理实现可以包括但不限于物理器件,而物理器件可以包括但不限于晶体管或忆阻器等器件。鉴于此,本文所述的各类装置(例如计算装置或其他处理装置)可以通过适当的硬件处理器来实现,例如中央处理器、GPU、FPGA、DSP和ASIC等。进一步,前述的所述存储单元或存储装置可以是任意适当的存储介质(包括磁存储 介质或磁光存储介质等),其例如可以是可变电阻式存储器(Resistive Random Access Memory,RRAM)、动态随机存取存储器(Dynamic Random Access Memory,DRAM)、静态随机存取存储器(Static Random Access Memory,SRAM)、增强动态随机存取存储器(Enhanced Dynamic Random Access Memory,EDRAM)、高带宽存储器(High Bandwidth Memory,HBM)、混合存储器立方体(Hybrid Memory Cube,HMC)、ROM和RAM等。
依据以下条款可更好地理解前述内容:
条款A1、一种计算装置,配置用于执行卷积运算,所述计算装置包括:
主处理电路,所述主处理电路用于:获取输入特征图和/或卷积核,其中所述输入特征图和卷积核已分别按卷积拆分方案拆分成多个拆分单元并转换其维度存储顺序,其中所述卷积拆分方案是根据所述输入特征图拆分前的最低存储维度的大小确定的,所述卷积拆分方案指示所述拆分单元的形状,一个拆分单元包含的数据量不超过硬件单次最大运算量,并且一个拆分单元内的数据连续存储为一个数据行;以及
多个从处理电路,所述多个从处理电路用于对所述输入特征图和卷积核的对应拆分单元执行卷积运算。
条款A2、根据条款A1所述的计算装置,其中所述卷积拆分方案是按如下确定的:
存在多个最近的M/4
n的倍数时,取M/4
n中最大值作为所述Uci,或者取其中对齐填补量最小的M/4
n作为所述Uci;以及
确定所述拆分单元在X和Y存储维度上的大小Ux和Uy,使得Uci×Ux×Uy=M,其中Ux=Uy。
条款A3、根据条款A1所述的计算装置,还包括分块电路,用于分别针对所述输入特征图和卷积核,按如下方式进行拆分和存储:
从以第一维度存储顺序存储的待运算数据中,以所述拆分单元为单位,按第一读取顺序读取一个或多个拆分单元,将读取的拆分单元存储到对应的存储电路上,其中每个拆分单元内的数据按照第二维度存储顺序存储,拆分单元之间按照第三维度存储顺序存储。
条款A4、根据条款A3所述的计算装置,其中:
所述第一维度存储顺序按照从高到低为HWC;
所述第二维度存储顺序按照从高到低为CHW;
所述第一读取顺序按照从高到低为HWC;
所述第三维度存储顺序与所述第一维度存储顺序相同;
其中H是高度维度、W是宽度维度、C是通道维度。
条款A5、根据条款A1-A4任一所述的计算装置,其中所述主处理电路进一步用于:
基于卷积核的输出通道Co维度尺寸和可调度的从处理电路数量Ns,确定完成所述卷积运算所需的运算轮次、各轮次运算中处理的Co数量或相应的分组模式。
条款A6、根据条款A5所述的计算装置,其中所述分组模式为GroupN,表示当前轮次运算中调度的所有从处理电路分为N组,每个从处理电路组处理同一Co值,不同从处理电路组处理不同Co值,N=4
n,n=0,1,2…。
条款A7、根据条款A6所述的计算装置,其中每个从处理电路组包括Rs个从处理电路,并且所述主处理电路进一步用于在所述Rs个从处理电路之间按如下划分所述输入特征图:
根据对应的输出特征图的尺寸,将所述输出特征图在HW维度上平均划分为Rs个形状相同的输出特征块;以及
根据计算每个输出特征块所需的输入特征图区域,将所述输入特征图在HW维度上划分为Rs个输入特征块,以分配给所述Rs个从处理电路。
条款A8、根据条款A7所述的计算装置,其中划分的所述输入特征块在HW维度上按所述拆分单元的YX维度对齐。
条款A9、根据条款A7-A8任一所述的计算装置,还包括第一存储电路和第二存储电路,
所述输入特征图和卷积核中之一确定为多播数据,拆分后的多播数据存储在第一存储电路中;以及
所述输入特征图和卷积核中另一确定为分发数据,拆分后的分发数据存储在第二存储电路中。
条款A10、根据条款A9所述的计算装置,其中所述第二存储电路包括为每个从处理电路分配的存储区域,
为每个从处理电路划分的输入特征图存储在所述第二存储电路中对应的存储区域内;或者
为每个从处理电路分配的卷积核存储在所述第二存储电路中对应的存储区域内。
条款A11、根据条款A9-A10任一所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存所述从处理电路对应的、来自所述第一存储电路和第二存储电路之一的多个输入特征行;
所述第二缓冲电路用于缓存所述从处理电路对应的、来自所述第一存储电路和第二存储电路另一的多个权值行;并且
每个运算电路用于在每次计算时,针对分别从所述第一缓冲电路中选取的输入特征行和从所述第二缓冲电路中选取的权值行执行对位乘累加运算。
条款A12、根据条款A11所述的计算装置,其中每个所述从处理电路进一步用于:
根据所述多个运算电路之间的输出点划分方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给所述从处理电路内的N
CU个运算电路以供计算;
从所述第二缓冲电路中滑动选取对应的权值数据,广播给所述N
CU个运算电路以供计算;以及
执行Nk次滑动选数,其中Nk根据卷积核在X和Y维度的尺寸和从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值来确定。
条款A13、根据条款A12所述的计算装置,其中当所述卷积运算为三维卷积运算时,所述从处理电路进一步用于按如下选取对应的权值数据:
从所述第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,将其复制Nop-1份扩展为一个扩展权值行,广播给所述从处理电路内的N
CU个运算电路,其中Nop是每个运算电路单次最大可计算卷积输出点数量。
条款A14、根据条款A13所述的计算装置,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值数据行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和;以及
将Nk次滑动计算中计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到Nop个运算结果。
条款A15、根据条款A12-A14任一所述的计算装置,其中每个所述从处理电路进一步用于:
根据所述多个运算电路之间的输出点划分方式,按特定顺序输出其内多个运算电路计算的输出点,以使得连续输出的输出点在X和/或Y维度上连续。
条款A16、根据条款A12-A15任一所述的计算装置,其中所述多个运算电路之间的输出点划分方式包括以下任一:
在每次计算时,每个运算电路计算在X和/或Y维度上连续的多个输出点;或者
每个运算电路计算在X和/或Y维度上间隔的多个输出点。
条款A17、根据条款A3所述的计算装置,其中所述分块电路进一步用于:
将从所述从处理电路返回的运算结果以第四维度存储顺序存储;以及
将所述运算结果转换为期望的维度存储顺序。
条款A18、根据条款A3或A17所述的计算装置,其中:
所述分块电路集成在所述主处理电路中;或者
所述分块电路独立于所述主处理电路。
条款A19、根据条款A3、A17或A18所述的计算装置,其中
所述分块电路对所述输入特征图和卷积核均执行所述拆分;或者
所述分块电路仅对所述输入特征图和卷积核中被确定为多播数据的数据执行所述拆分。
条款A20、一种芯片,包括根据条款A1-A19任一所述的计算装置。
条款A21、一种板卡,包括根据条款A20所述的芯片。
条款A22、一种利用条款A1-A19任一所述的计算装置执行卷积运算的方法。
条款B1、一种用于执行卷积运算的处理电路,包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存待运算的多个输入特征行;
所述第二缓冲电路用于缓存待运算的多个权值行;以及
每个所述运算电路用于在每次计算时,针对分别从所述第一缓冲电路中选取的输入特征行和从所述第二缓冲电路中选取的权值行执行对位乘累加运算,其中所述权值行是根据从所述第二缓冲电路中选取的局部权值行复制扩展而成的扩展权值行。
条款B2、根据条款B1所述的处理电路,其中所述处理电路进一步用于在每次滑动选数时:
根据所述多个运算电路之间的输出点划分方式,从所述第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给所述处理电路内的N
CU个运算电路以供计算;以及
从所述第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,将其复制Nop-1份扩展为一个扩展权值行,广播给所述N
CU个运算电路,其中Nop是每个运算电路单次最大可计算卷积输出点数量。
条款B3、根据条款B2所述的处理电路,其中所述处理电路进一步用于:
执行Nk次滑动选数,其中Nk根据卷积核在X和Y维度的尺寸和所述处理电路在当前卷积运算模式下单次运算所支持的最大卷积核尺寸中的较小值来确定。
条款B4、根据条款B3所述的处理电路,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值行,以1/Nop个行为单位进行对位乘累加,得到Nop个部分和;以及
将Nk次滑动计算中计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到Nop个运算结果。
条款B5、根据条款B4所述的处理电路,其中所述处理电路进一步用于:
根据所述多个运算电路之间的输出点划分方式,按特定顺序输出其内多个运算电路计算的输出点,以使得连续输出的输出点在X和/或Y维度上连续。
条款B6、根据条款B2-B5任一所述的处理电路,其中所述多个运算电路之间的输出点划分方式包括以下任一:
在每次计算时,每个运算电路计算在X和/或Y维度上连续的多个输出点;或者
在每次计算时,每个运算电路计算在X和/或Y维度上间隔的多个输出点。
条款B7、根据条款B2-B6任一所述的处理电路,其中N
CU=4,Nop=4。
条款B8、根据条款B1-B7任一所述的处理电路,其中每个所述输入特征行和所述权值行均由一个拆分单元构成,一个所述拆分单元包含最低存储维度和至少一个其他存储维度的数据。
条款B9、根据条款B8所述的处理电路,其中所述拆分单元的形状为Uci×Ux×Uy=M,Uci为所述拆分单元在所述输入特征数据和权值数据初始的最低存储维度上的尺寸,Ux和Uy分别为所述拆分单元在所述输入特征数据和权值数据初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量,Uci=M/4
n,
条款B10、一种计算装置,配置用于执行卷积运算,所述计算装置包括主处理电路和多个从处理电路,每个所述从处理电路配置成根据条款B1-B9任一所述的处理电路。
条款B11、一种芯片,包括根据条款B10所述的计算装置。
条款B12、一种板卡,包括根据条款B11所述的芯片。
条款B13、一种利用条款B1-B9任一所述的处理电路执行卷积运算的方法。
条款C1、一种计算装置,配置用于执行卷积运算,所述计算装置包括:
分块电路,用于按照卷积拆分方案,将输入特征图和卷积核拆分成多个对应的拆分单元,其中一个拆分单元包括最低存储维度和至少一个其他存储维度的数据,并且一个拆分单元的数据量不超过硬件单次最大运算量;以及转换所述输入特征图和卷积核的维度存储顺序,以使得一个拆分单元内的数据连续存储为一个数据行,其中拆分和转换后的输入特征图和/或卷积核被提供给主处理电路或从处理电路;
所述主处理电路,用于将其获得的数据分发给多个从处理电路以供执行卷积运算;以及根据所述卷积拆分方案,对所述多个从处理电路返回的运算结果进行拼接处理,以得到所述输入特征图和卷积核的卷积运算的输出特征图;以及
所述多个从处理电路,用于根据其获得的数据执行卷积运算,并向所述主处理电路返回运算结果。
条款C2、根据条款C1所述的计算装置,其中所述卷积拆分方案还指示执行所述卷积运算的运算轮次,其中每个运算轮次中处理的输出通道Co数量对应于该运算轮次中可调度的从处理电路数量Ns。
条款C3、根据条款C2所述的计算装置,其中所述计算装置还包括第一存储电路和第二存储电路,
所述输入特征图确定为多播数据,拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线传输给所调度的多个从处理电路;以及
所述卷积核确定为分发数据,拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以在运算前分发给对应的从处理电路。
条款C4、根据条款C3所述的计算装置,其中各个运算轮次中分配给各个从处理电路的、不同Co值的卷积核分别存储在所述第二存储电路中为对应的从处理电路分配的存储区域中。
条款C5、根据条款C3-C4任一所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存来自所述第一存储电路的、广播传输的多个输入特征数据行;
所述第二缓冲电路用于缓存来自所述第二存储电路的、分发给所述从处理电路的卷积核的多个权值数据行;并且
每个运算电路用于在每次运算中,针对分别从所述第一缓冲电路中选取的输入特征数据行和从所述第二缓冲电路中选取的权值数据行执行对位乘累加运算。
条款C6、根据条款C5所述的计算装置,其中所述从处理电路进一步用于按如下在其可调度的N
CU个运算电路之间划分输出点:
在每次计算时,每个运算电路计算所述输出特征图上在X和/或Y维度连续的多个输出点。
条款C7、根据条款C6所述的计算装置,其中所述卷积运算为三维卷积运算,并且每个所述从处理电路进一步用于:
按照与所述输出点的划分方式对应的方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给所述N
CU个运算电路以供计算;
从所述第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,其中Nop是每个运算电路单次最大可计算卷积输出点数量,将其复制Nop-1份扩展为一个扩展权值行,广播给所述从处理电路内的N
CU个运算电路;以及
执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是卷积核在X和Y维度的尺寸或所述从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值。
条款C8、根据条款C7所述的计算装置,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和;以及
将Nk次滑动计算中计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到Nop个运算结果。
条款C9、根据条款C8所述的计算装置,其中每个所述从处理电路进一步用于:
按照输出点连续划分的顺序,每次输出其内一个运算电路的Nop个运算结果。
条款C10、根据条款C5-C9任一所述的计算装置,其中所述从处理电路进一步用于:
根据所述运算电路内的存储空间限制,确定从处理电路内的权值复用次数rs;以及
控制所述第一缓冲电路中的输入特征数据的加载频次,以使得第二缓冲电路中每次加载的权值数据重复使用rs次,与所述第一缓冲电路中rs次加载的对应输入特征数据执行卷积运算。
条款C11、根据条款C1-C10任一所述的计算装置,其中所述卷积拆分方案指示所述拆分单元的形状为Uci×Ux×Uy=M,Uci为所述拆分单元在所述输入特征图和卷积核初始的最低存储维度上的尺寸,Ux和Uy分别为所述拆分单元在所述输入特征图和卷积核初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量,Uci>Ux=Uy>1,Uci=M/4
n,
条款C12、根据条款C11所述的计算装置,其中M=64字节,Uci=16字节,Ux=Uy=2。
条款C13、根据条款C6所述的计算装置,其中在每次计算时,每个运算电路计算包括2×2个输出点的输出特征区块。
条款C14、根据条款C7所述的计算装置,其中N
CU=4,Nop=4。
条款C15、根据条款C7所述的计算装置,其中所述从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸为3×3。
条款C16、一种芯片,包括根据条款C1-C15任一所述的计算装置。
条款C17、一种板卡,包括根据条款C16所述的芯片。
条款C18、一种利用条款C1-C15任一所述的计算装置执行卷积运算的方法。
条款D1、一种计算装置,配置用于执行卷积运算,所述计算装置包括:
主处理电路,所述主处理电路用于:获取输入特征图和/或卷积核,其中所述输入特征图和卷积核已分别按卷积拆分方案拆分成多个拆分单元并转换维度存储顺序,其中一个拆分单元包括最低存储维度和至少一个其他存储维度的数据,并且一个拆分单元的数据量不超过硬件单次最大运算量,单轮运算中卷积核的输出通道Co维度的尺寸不超过所述从处理电路的数量,并且一个拆分单元内的数据连续存储为一个数据行;以及
多个从处理电路,所述多个从处理电路用于对所述输入特征图和卷积核的对应数据行执行卷积运算。
条款D2、根据条款D1所述的计算装置,其中所述卷积拆分方案还指示执行所述卷积运算的运算轮次、各轮次运算中处理的Co数量及相应的分组模式。
条款D3、根据条款D2所述的计算装置,其中所述分组模式为GroupN,表示当前轮次运算中执行运算的Ns个从处理电路分为N组,每个从处理电路组处理同一Co值,不同从处理电路组处理不同Co值,N=4
n,n=0,1,2…。
条款D4、根据条款D3所述的计算装置,其中每个从处理电路组包括Rs个从处理电路,并且所述主处理电路进一步用于在所述Rs个从处理电路之间按如下划分所述输入特征图:
根据对应的输出特征图的尺寸,将所述输出特征图在HW维度上平均划分为Rs个形状相同的输出特征块;以及
根据计算每个输出特征块所需的输入特征图区域,将所述输入特征图在HW维度上划分为Rs个输入特征块,以分配给所述Rs个从处理电路。
条款D5、根据条款D4所述的计算装置,其中所述计算装置还包括第一存储电路和第二存储电路,
所述卷积核确定为多播数据,拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线传输给所调度的多个从处理电路;以及
所述输入特征图确定为分发数据,拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以便分发给对应的从处理电路。
条款D6、根据条款D5所述的计算装置,其中所述Rs个输入特征块分别按所述拆分单元进行拆分并转换维度存储顺序后存储在所述第二存储电路中为所述Rs个从处理电路分配的存储区域中。
条款D7、根据条款D5-D6任一所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存来自所述第二存储电路的、分发给所述从处理电路的多个输入特征数据行;
所述第二缓冲电路用于缓存来自所述第一存储电路的、多播传输给所述从处理电路的对应输出通道值的卷积核的多个权值数据行;并且
每个运算电路用于在每次运算中,针对分别从所述第一缓冲电路中选取的输入特征数据行和从所述第二缓冲电路中选取的权值数据行执行对位乘累加运算。
条款D8、根据条款D7所述的计算装置,其中所述从处理电路进一步用于按如下在其可调度的N
CU个运算电路之间划分输出点:
每次计算时,每个运算电路计算所述输出特征图上在X和/或Y维度间隔的多个输出点。
条款D9、根据条款D8所述的计算装置,其中所述卷积运算为三维卷积运算,并且每个所述从处理电路进一步用于:
按照与所述输出点的划分方式对应的方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给所述从处理电路内的N
CU个运算电路以供计算;
从所述第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值数据行,将其复制Nop-1份扩展为一个扩展权值行,广播给所述从处理电路内的N
CU个运算电路;以及
执行Nk次滑动选数,其中Nk=ceil(Kx/2)*ceil(Ky/2),Kx和Ky分别是卷积核在X和Y维度的尺寸或所述从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值。
条款D10、根据条款D9所述的计算装置,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和;以及
将Nk次滑动计算中计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到Nop个运算结果。
条款D11、根据条款D10所述的计算装置,其中每个所述从处理电路进一步用于:
每次输出其内部分运算电路的部分运算结果,所述部分运算结果在输出特征图的X和/或Y维度上连续。
条款D12、根据条款D7-D11任一所述的计算装置,其中所述从处理电路进一步用于:
根据所述运算电路内的存储空间限制,确定从处理电路内的输入特征复用次数rn;以及
控制所述第二缓冲电路中的权值数据的加载频次,以使得第一缓冲电路中每次加载的输入特征数据重复使用rn次,与所述第二缓冲电路中rn次加载的对应权值数据执行卷积运算。
条款D13、根据条款D1-D12任一所述的计算装置,其中所述卷积拆分方案指示所述拆分单元的尺寸为Uci×Uy×Ux=M,Uci为所述拆分单元在所述输入特征图和卷积核初始的最低存储维度上的尺寸,Ux和Uy分别为所述拆分单元在所述输入特征图和卷积核初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量,Ux=Uy≥Uci>1,Uci=M/4
n,
条款D14、根据条款D13所述的计算装置,其中M=64字节,Uci=4字节,Ux=Uy=4。
条款D15、根据条款D8所述的计算装置,其中在每次计算时,每个运算电路计算X和Y维度上均间隔1的2×2个输出点。
条款D16、根据条款D9所述的计算装置,其中N
CU=4,Nop=4。
条款D17、根据条款D9所述的计算装置,其中所述从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸为8×8。
条款D18、一种芯片,包括根据条款D1-D17任一所述的计算装置。
条款D19、一种板卡,包括根据条款D18所述的芯片。
条款D20、一种利用条款D1-D17任一所述的计算装置执行卷积运算的方法。
条款E1、一种计算装置,配置用于执行深度卷积运算,所述计算装置包括:
主处理电路,所述主处理电路用于:获取输入特征图和/或卷积核,其中所述输入特征图和卷 积核已分别按卷积拆分方案拆分成多个拆分单元并转换维度存储顺序,其中一个拆分单元包括最低存储维度和至少一个其他存储维度的数据,并且一个拆分单元的总数据量不超过硬件单次最大运算量,并且一个拆分单元内的数据连续存储为一个数据行;以及
多个从处理电路,所述多个从处理电路用于对所述输入特征图和卷积核的对应数据行执行深度卷积运算。
条款E2、根据条款E1所述的计算装置,其中所述卷积拆分方案指示所述拆分单元的形状为Uc×Uy×Ux=M,Uc为所述拆分单元在所述输入特征图和卷积核初始的最低存储维度C上的尺寸,Ux和Uy分别为所述拆分单元在所述输入特征图和卷积核初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量,Ux=Uy≥Uc>1,Uc=M/4
n,
条款E3、根据条款E2所述的计算装置,其中所述卷积拆分方案还指示执行所述卷积运算的运算轮次、各轮次运算中处理的C数量Nc及相应的分组模式,其中Nc对齐到Uc。
条款E4、根据条款E3所述的计算装置,其中所述分组模式为GroupN,表示当前轮次运算中调度的Ns个从处理电路分为N组,每个从处理电路组处理相同的连续Uc个C值,不同从处理电路组处理不同的连续Uc个C值,N=4
n,n=0,1,2…。
条款E5、根据条款E4所述的计算装置,其中每个从处理电路组包括Rs个从处理电路,并且所述主处理电路进一步用于在所述Rs个从处理电路之间按如下划分所述输入特征图:
根据对应的输出特征图的尺寸,将所述输出特征图在HW维度上平均划分为Rs个形状相同的输出特征块;以及
根据计算每个输出特征块所需的输入特征图区域,将所述输入特征图在HW维度上划分为Rs个输入特征块,以分配给所述Rs个从处理电路。
条款E6、根据条款E5所述的计算装置,其中所述计算装置还包括第一存储电路和第二存储电路,
所述卷积核确定为多播数据,拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线传输给所调度的多个从处理电路;以及
所述输入特征图确定为分发数据,拆分并转换维度存储顺序后的分发数据存储在第二存储电路中,以便分发给对应的从处理电路。
条款E7、根据条款E6所述的计算装置,其中
所述Rs个输入特征块分别按所述拆分单元进行拆分并转换维度存储顺序后存储在所述第二存储电路中为所述Rs个从处理电路分配的存储区域中。
条款E8、根据条款E6-E7任一所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存来自所述第二存储电路的、分发给所述从处理电路的多个输入特征数据行;
所述第二缓冲电路用于缓存来自所述第一存储电路的、多播传输给所述从处理电路的多个权值数据行;并且
每个运算电路用于在每次运算中,针对分别从所述第一缓冲电路中选取的输入特征数据行和从所述第二缓冲电路中选取的权值数据行执行对位乘累加运算。
条款E9、根据条款E8所述的计算装置,其中所述从处理电路进一步用于按如下在其可调度的N
CU个运算电路之间划分输出点:
在每次计算时,每个运算电路计算所述输出特征图上在X和/或Y维度间隔的1个输出点;以及
在不同次计算中,每个运算电路计算所述输出特征图上在X和/或Y维度上不同的输出点。
条款E10、根据条款E9所述的计算装置,其中每个所述从处理电路进一步用于:
按照与所述输出点的划分方式对应的方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取N
CU个输入特征行,分别发送给所述从处理电路内的N
CU个运算电路以供计算;
从所述第二缓冲电路中读取1个权值数据行,广播给所述从处理电路内的N
CU个运算电路; 以及
在所述第一缓冲电路上执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是卷积核在X和Y维度的尺寸或所述从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值。
条款E11、根据条款E10所述的计算装置,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个权值行,以1/Uc个数据行为单位,将对应同一通道值的特征数据和权值数据进行对位乘累加,得到Uc个输出点;以及
将Nk次滑动计算中计算得到的Nk*Uc个输出点按照所述输出点的划分方式进行拼接,得到Uc个通道上的Nk*N
CU个运算结果。
条款E12、根据条款E11所述的计算装置,其中每个所述从处理电路进一步用于:
每次输出其内部分运算电路的部分运算结果,所述部分运算结果在输出特征图的X和/或Y维度上连续。
条款E13、根据条款E8-E12任一所述的计算装置,其中所述从处理电路进一步用于:
根据所述运算电路内的存储空间限制,确定从处理电路内的输入特征复用次数rn;以及
控制所述第二缓冲电路中的权值数据的加载频次,以使得第一缓冲电路中每次加载的输入特征数据重复使用rn次,与所述第二缓冲电路中rn次加载的对应权值数据执行卷积运算。
条款E14、根据条款E2-E13任一所述的计算装置,其中M=64字节,Uci=4字节,Ux=Uy=4。
条款E15、根据条款E10所述的计算装置,其中N
CU=4,Nop=4。
条款E16、根据条款E10所述的计算装置,其中所述从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸为4×4。
条款E17、一种芯片,包括根据条款E1-E16任一所述的计算装置。
条款E18、一种板卡,包括根据条款E17所述的芯片。
条款E19、一种利用条款E1-E16任一所述的计算装置执行卷积运算的方法。
条款F1、一种计算装置,配置用于执行神经网络模型的反向训练中的深度卷积运算,所述计算装置包括:
主处理电路,所述主处理电路用于:获取输入神经元数据和/或神经元梯度数据,其中所述输入神经元数据和神经元梯度数据已分别按卷积拆分方案拆分成多个拆分单元并转换维度存储顺序,其中一个拆分单元包括最低存储维度和至少一个其他存储维度的数据,并且一个拆分单元的总数据量不超过硬件单次最大运算量,并且一个拆分单元内的数据连续存储为一个数据行;以及
多个从处理电路,所述多个从处理电路用于对所述输入神经元数据和神经元梯度数据的对应数据行执行所述深度卷积运算。
条款F2、根据条款F1所述的计算装置,其中所述卷积拆分方案指示所述拆分单元的形状为Uc×Uy×Ux=M,Uc为所述拆分单元在所述输入神经元数据和神经元梯度数据初始的最低存储维度、通道C上的尺寸,Ux和Uy分别为所述拆分单元在所述输入神经元数据和神经元梯度数据初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量,Ux=Uy≥Uc>1,Uc=M/4
n,
条款F3、根据条款F2所述的计算装置,其中所述卷积拆分方案还指示执行所述深度卷积运算的分组划分方式,其中所述分组划分方式针对所述输入神经元数据和神经元梯度数据,按照通道C维度、以Uc为单位顺次划分至可调度的Ns个从处理电路,每个从处理电路处理不同的连续Uc个C值的输入神经元数据和神经元梯度数据。
条款F4、根据条款F3所述的计算装置,其中所述计算装置还包括第一存储电路和第二存储电路,
所述神经元梯度数据确定为单播数据,拆分并转换维度存储顺序后的单播数据存储在第一存储电路中,以在运算期间通过广播总线将对应不同Uc个C值的神经元梯度数据分别传输给所调度的Ns个从处理电路;以及
所述输入神经元数据确定为分发数据,拆分并转换维度存储顺序后的分发数据按照通道C维度、以Uc为单位顺次划分的方式分别存储在第二存储电路中与所述Ns个从处理电路对应的存储区域内,以便分发给对应的从处理电路。
条款F5、根据条款F4所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存来自所述第二存储电路的、分发给所述从处理电路的多个输入神经元数据行;
所述第二缓冲电路用于缓存来自所述第一存储电路的、单播传输给所述从处理电路的多个神经元梯度数据行;并且
每个运算电路用于在每次运算中,针对分别从所述第一缓冲电路中选取的输入神经元数据行和从所述第二缓冲电路中选取的神经元梯度数据行执行对位乘累加运算。
条款F6、根据条款F5所述的计算装置,其中所述从处理电路进一步用于按如下在其可调度的N
CU个运算电路之间划分输出点:
在每次计算时,每个运算电路计算所述权值梯度数据的Uc个通道C值的XY面上、在X和/或Y维度相邻的1个输出点;以及
在不同次计算中,每个运算电路计算所述权值梯度数据在X和/或Y维度上不同的输出点。
条款F7、根据条款F6所述的计算装置,其中每个所述从处理电路进一步用于:
按照与所述输出点的划分方式对应的方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取N
CU个输入神经元数据行,分别发送给所述从处理电路内的N
CU个运算电路以供计算;
从所述第二缓冲电路中读取1个神经元梯度数据行,广播给所述从处理电路内的N
CU个运算电路;以及
在所述第一缓冲电路上执行Nk次滑动选数,其中Nk=ceil(Kx/2)*ceil(Ky/2),Kx和Ky分别是权值梯度数据在X和Y维度的尺寸或所述从处理电路在所述卷积拆分模式下单次运算所支持的最大权值梯度尺寸中的较小值。
条款F8、根据条款F7所述的计算装置,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入神经元数据行和来自第二缓冲电路的一个神经元梯度数据行,以1/Uc个数据行为单位,将对应同一通道值的输入神经元数据和神经元梯度数据进行对位乘累加,得到Uc个XY面上同一位置的1个输出点;以及
在Nk次滑动计算中,计算得到Uc个XY面上X和/或Y维度上间隔的Nk个输出点。
条款F9、根据条款F8所述的计算装置,其中每个所述从处理电路进一步用于:
每次输出其内一个运算电路运算得到的Uc个XY面上同一位置的1个输出点。
条款F10、根据条款F9所述的计算装置,其中所述主处理电路进一步用于:
将从所述从处理电路输出的运算结果按照Ky*Kx*(Ns*Uc)的维度顺序进行拼接存储。
条款F11、根据条款F1-F10任一所述的计算装置,其中M=64字节,Uc=4字节,Ux=Uy=4。
条款F12、根据条款F6-F10任一所述的计算装置,其中N
CU=4,Ns=16。
条款F13、根据条款F7所述的计算装置,其中所述从处理电路在所述卷积拆分模式下单次运算所支持的最大权值梯度尺寸为4×4。
条款F14、一种芯片,包括根据条款F1-F13任一所述的计算装置。
条款F15、一种板卡,包括根据条款F14所述的芯片。
条款F16、一种利用条款F1-F13任一所述的计算装置执行卷积运算的方法。
条款G1、一种计算装置,配置用于执行神经网络模型的反向训练中的叉乘卷积运算,所述计算装置包括:
主处理电路,所述主处理电路用于:获取输入神经元数据和/或神经元梯度数据,其中所述输入神经元数据和神经元梯度数据已分别按卷积拆分方案拆分成多个拆分单元,其中一个拆分单元包括最低存储维度和至少一个其他存储维度的数据,并且一个拆分单元的总数据量不超过硬件单 次最大运算量,并且一个拆分单元内的数据连续存储为一个数据行;以及
多个从处理电路,所述多个从处理电路用于对所述输入神经元数据和神经元梯度数据的对应数据行执行所述叉乘卷积运算。
条款G2、根据条款G1所述的计算装置,其中所述卷积拆分方案指示所述拆分单元的形状为Uc×Uy×Ux=M,Uc为所述拆分单元在所述输入神经元数据初始的最低存储维度、输入通道Ci上的尺寸和在所述神经元梯度数据初始的最低存储维度、输出通道Co上的尺寸,Ux和Uy分别为所述拆分单元在所述输入神经元数据和神经元梯度数据初始的X和Y存储维度上的尺寸,M是硬件单次最大运算量,Ux=Uy≥Uc>1,Uc=M/4
n,
条款G3、根据条款G2所述的计算装置,其中所述卷积拆分方案还指示执行所述深度卷积运算的运算轮次、各轮次运算中处理的输出通道Co数量Nco以及相应的分组模式,其中Nco对齐到Uc。
条款G4、根据条款G3所述的计算装置,其中所述分组模式为GroupN,表示当前轮次运算中调度的Ns个从处理电路分为N组,每个从处理电路组处理相同的连续Uc个Co值,不同从处理电路组处理不同的连续Uc个Co值,N=4
n,n=0,1,2…。
条款G5、根据条款G4所述的计算装置,其中所述卷积拆分方案还指示在每个从处理电路组内,针对所述输入神经元数据,按照输入通道Ci维度、以Uc为单位顺次划分至同一组内可调度的Rs个从处理电路,其中Rs=Ns/N。
条款G6、根据条款G5所述的计算装置,其中所述计算装置还包括第一存储电路和第二存储电路,
所述神经元梯度数据确定为多播数据,拆分并转换维度存储顺序后的多播数据存储在第一存储电路中,以在运算期间通过广播总线将对应不同Uc个Co值的神经元梯度数据分别传输给所调度的N个从处理电路组,每个从处理电路组共享相同的Uc个Co值的神经元梯度数据;以及
所述输入神经元数据确定为分发数据,拆分并转换维度存储顺序后的分发数据复制N份,每份按照Ci方向、以Uc为单位顺次划分的方式划分为Rs个数据块,分别存储在第二存储电路中对应的存储区域内,以便分发给对应的从处理电路。
条款G7、根据条款G6所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:
所述第一缓冲电路用于缓存来自所述第二存储电路的、分发给所述从处理电路的多个输入神经元数据行;
所述第二缓冲电路用于缓存来自所述第一存储电路的、多播传输给所述从处理电路的多个神经元梯度数据行;并且
每个运算电路用于在每次运算中,针对分别从所述第一缓冲电路中选取的输入神经元数据行和从所述第二缓冲电路中选取的神经元梯度数据行执行对位乘累加运算。
条款G8、根据条款G7所述的计算装置,其中所述从处理电路进一步用于按如下在其可调度的N
CU个运算电路之间划分输出点:
每次计算时,每个运算电路计算所述权值梯度数据在不同Co上、连续Uc个Ci上、XY维度上同一位置的1个输出点;以及
在不同次计算中,每个运算电路计算所述权值梯度数据在X和/或Y维度上不同的输出点。
条款G9、根据条款G8所述的计算装置,其中每个所述从处理电路进一步用于:
按照与所述输出点的划分方式对应的方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取1个输入神经元数据行,广播传输给所述从处理电路内的N
CU个运算电路以供计算;
从所述第二缓冲电路中读取1个神经元梯度数据行,按照Co维度拆分成Uc个Co值,每个Co值对应的XY数据面复制Uc份,分别发送给所述从处理电路内的Uc个运算电路;以及
在所述第一缓冲电路上执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是权值梯度数据在X和Y维度的尺寸或所述从处理电路在所述卷积拆分模式下单次运算所支持的最大权值梯度尺寸中的较小值。
条款G10、根据条款G8所述的计算装置,其中当Uc<N
CU时,每个所述从处理电路进一步用于:
按照与所述输出点的划分方式对应的方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取1个输入神经元数据行,广播传输给所述从处理电路内的N
CU个运算电路以供计算;
从所述第二缓冲电路中读取N
CU/Uc个神经元梯度数据行,按照Co维度拆分成N
CU个Co值,每个Co值对应的XY数据面复制Uc份,分别发送给所述从处理电路内的N
CU个运算电路;以及
在所述第一缓冲电路上执行Nk次滑动选数,其中Nk=Kx*Ky,Kx和Ky分别是权值梯度数据在X和Y维度的尺寸或所述从处理电路在所述卷积拆分模式下单次运算所支持的最大权值梯度尺寸中的较小值。
条款G11、根据条款G9或G10所述的计算装置,其中每个所述运算电路进一步用于:
在每次计算时,针对来自第一缓冲电路的一个输入神经元数据行和来自第二缓冲电路的一个神经元梯度数据行,以1/Uc个数据行为单位,将对应同一输入通道Ci值的输入神经元数据和神经元梯度数据进行对位乘累加,得到所分配Co值在Ci维度上的Uc个输出点;以及
在Nk次滑动计算中,计算得到Nk*Uc个输出点,其为单个Co上、Uc个Ci上的XY面上X和/或Y维度上连续的Nk个输出点。
条款G12、根据条款G11所述的计算装置,其中每个所述从处理电路进一步用于:
每次输出其内一个运算电路运算得到的一个Co上、Uc个Ci上的XY面上同一位置的1个输出点。
条款G13、根据条款G12所述的计算装置,其中所述主处理电路进一步用于:
将所有从处理电路输出的运算结果按照Ky*Kx*Co/N*N*(Rs*Uc)的维度顺序进行拼接存储,其中N是分组数。
条款G14、根据条款G1-G13任一所述的计算装置,其中M=64字节,Uc=4字节,Ux=Uy=4。
条款G15、根据条款G8-G13任一所述的计算装置,其中N
CU=4,Ns=16。
条款G16、根据条款G9-G10任一所述的计算装置,其中所述从处理电路在所述卷积拆分模式下单次运算所支持的最大权值梯度尺寸为4×4。
条款G17、一种芯片,包括根据条款G1-G16任一所述的计算装置。
条款G18、一种板卡,包括根据条款G17所述的芯片。
条款G19、一种利用条款G1-G16任一所述的计算装置执行卷积运算的方法。
以上对本披露实施例进行了详细介绍,本文中应用了具体个例对本披露的原理及实施方式进行了阐述,以上实施例的说明只是用于帮助理解本披露的方法及其核心思想;同时,对于本领域的一般技术人员,依据本披露的思想,在具体实施方式及应用范围上均会有改变之处,综上所述,本说明书内容不应理解为对本披露的限制。
Claims (22)
- 一种计算装置,配置用于执行卷积运算,所述计算装置包括:主处理电路,所述主处理电路用于:获取输入特征图和/或卷积核,其中所述输入特征图和卷积核已分别按卷积拆分方案拆分成多个拆分单元并转换其维度存储顺序,其中所述卷积拆分方案是根据所述输入特征图拆分前的最低存储维度的大小确定的,所述卷积拆分方案指示所述拆分单元的形状,一个拆分单元包含的数据量不超过硬件单次最大运算量,并且一个拆分单元内的数据连续存储为一个数据行;以及多个从处理电路,所述多个从处理电路用于对所述输入特征图和卷积核的对应拆分单元执行卷积运算。
- 根据权利要求1所述的计算装置,还包括分块电路,用于分别针对所述输入特征图和卷积核,按如下方式进行拆分和存储:从以第一维度存储顺序存储的待运算数据中,以所述拆分单元为单位,按第一读取顺序读取一个或多个拆分单元,将读取的拆分单元存储到对应的存储电路上,其中每个拆分单元内的数据按照第二维度存储顺序存储,拆分单元之间按照第三维度存储顺序存储。
- 根据权利要求3所述的计算装置,其中:所述第一维度存储顺序按照从高到低为HWC;所述第二维度存储顺序按照从高到低为CHW;所述第一读取顺序按照从高到低为HWC;所述第三维度存储顺序与所述第一维度存储顺序相同;其中H是高度维度、W是宽度维度、C是通道维度。
- 根据权利要求1-4任一所述的计算装置,其中所述主处理电路进一步用于:基于卷积核的输出通道Co维度尺寸和可调度的从处理电路数量Ns,确定完成所述卷积运算所需的运算轮次、各轮次运算中处理的Co数量或相应的分组模式。
- 根据权利要求5所述的计算装置,其中所述分组模式为GroupN,表示当前轮次运算中调度的所有从处理电路分为N组,每个从处理电路组处理同一Co值,不同从处理电路组处理不同Co值,N=4 n,n=0,1,2…。
- 根据权利要求6所述的计算装置,其中每个从处理电路组包括Rs个从处理电路,并且所述主处理电路进一步用于在所述Rs个从处理电路之间按如下划分所述输入特征图:根据对应的输出特征图的尺寸,将所述输出特征图在HW维度上平均划分为Rs个形状相同的输出特征块;以及根据计算每个输出特征块所需的输入特征图区域,将所述输入特征图在HW维度上划分为Rs个输入特征块,以分配给所述Rs个从处理电路。
- 根据权利要求7所述的计算装置,其中划分的所述输入特征块在HW维度上按所述拆分单元的YX维度对齐。
- 根据权利要求7-8任一所述的计算装置,还包括第一存储电路和第二存储电路,所述输入特征图和卷积核中之一确定为多播数据,拆分后的多播数据存储在第一存储电路中;以及所述输入特征图和卷积核中另一确定为分发数据,拆分后的分发数据存储在第二存储电路中。
- 根据权利要求9所述的计算装置,其中所述第二存储电路包括为每个从处理电路分配的存储区域,为每个从处理电路划分的输入特征图存储在所述第二存储电路中对应的存储区域内;或者为每个从处理电路分配的卷积核存储在所述第二存储电路中对应的存储区域内。
- 根据权利要求9-10任一所述的计算装置,其中每个所述从处理电路包括第一缓冲电路、第二缓冲电路和多个运算电路,其中:所述第一缓冲电路用于缓存所述从处理电路对应的、来自所述第一存储电路和第二存储电路之一的多个输入特征行;所述第二缓冲电路用于缓存所述从处理电路对应的、来自所述第一存储电路和第二存储电路另一的多个权值行;并且每个运算电路用于在每次计算时,针对分别从所述第一缓冲电路中选取的输入特征行和从所述第二缓冲电路中选取的权值行执行对位乘累加运算。
- 根据权利要求11所述的计算装置,其中每个所述从处理电路进一步用于:根据所述多个运算电路之间的输出点划分方式,以所述拆分单元为滑动窗口,从所述第一缓冲电路中滑动选取N CU个输入特征行,分别发送给所述从处理电路内的N CU个运算电路以供计算;从所述第二缓冲电路中滑动选取对应的权值数据,广播给所述N CU个运算电路以供计算;以及执行Nk次滑动选数,其中Nk根据卷积核在X和Y维度的尺寸和从处理电路在所述卷积拆分模式下单次运算所支持的最大卷积核尺寸中的较小值来确定。
- 根据权利要求12所述的计算装置,其中当所述卷积运算为三维卷积运算时,所述从处理电路进一步用于按如下选取对应的权值数据:从所述第二缓冲电路中按照与在第一缓冲电路中对应的滑动方式选取1/Nop个权值行,将其复制Nop-1份扩展为一个扩展权值行,广播给所述从处理电路内的N CU个运算电路,其中Nop是每个运算电路单次最大可计算卷积输出点数量。
- 根据权利要求13所述的计算装置,其中每个所述运算电路进一步用于:在每次计算时,针对来自第一缓冲电路的一个输入特征行和来自第二缓冲电路的一个扩展权值数据行,以1/Nop个数据行为单位进行对位乘累加,得到Nop个部分和;以及将Nk次滑动计算中计算得到的Nk*Nop个部分和按照对应的卷积输出点进行累加,得到Nop个运算结果。
- 根据权利要求12-14任一所述的计算装置,其中每个所述从处理电路进一步用于:根据所述多个运算电路之间的输出点划分方式,按特定顺序输出其内多个运算电路计算的输出点,以使得连续输出的输出点在X和/或Y维度上连续。
- 根据权利要求12-15任一所述的计算装置,其中所述多个运算电路之间的输出点划分方式包括以下任一:在每次计算时,每个运算电路计算在X和/或Y维度上连续的多个输出点;或者每个运算电路计算在X和/或Y维度上间隔的多个输出点。
- 根据权利要求3所述的计算装置,其中所述分块电路进一步用于:将从所述从处理电路返回的运算结果以第四维度存储顺序存储;以及将所述运算结果转换为期望的维度存储顺序。
- 根据权利要求3或17所述的计算装置,其中:所述分块电路集成在所述主处理电路中;或者所述分块电路独立于所述主处理电路。
- 根据权利要求3、17或18所述的计算装置,其中所述分块电路对所述输入特征图和卷积核均执行所述拆分;或者所述分块电路仅对所述输入特征图和卷积核中被确定为多播数据的数据执行所述拆分。
- 一种芯片,包括根据权利要求1-19任一所述的计算装置。
- 一种板卡,包括根据权利要求20所述的芯片。
- 一种利用权利要求1-19任一所述的计算装置执行卷积运算的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111131388.5A CN115878547A (zh) | 2021-09-26 | 2021-09-26 | 计算装置、利用计算装置实施卷积运算的方法及相关产品 |
CN202111131388.5 | 2021-09-26 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023045638A1 true WO2023045638A1 (zh) | 2023-03-30 |
Family
ID=85720032
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/113302 WO2023045638A1 (zh) | 2021-09-26 | 2022-08-18 | 计算装置、利用计算装置实施卷积运算的方法及相关产品 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN115878547A (zh) |
WO (1) | WO2023045638A1 (zh) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107437110A (zh) * | 2017-07-11 | 2017-12-05 | 中国科学院自动化研究所 | 卷积神经网络的分块卷积优化方法及装置 |
CN109919311A (zh) * | 2019-03-13 | 2019-06-21 | 北京地平线机器人技术研发有限公司 | 生成指令序列的方法、执行神经网络运算的方法和装置 |
US20200167405A1 (en) * | 2018-11-28 | 2020-05-28 | Electronics And Telecommunications Research Institute | Convolutional operation device with dimensional conversion |
CN111738424A (zh) * | 2020-06-29 | 2020-10-02 | 湖南国科微电子股份有限公司 | 神经网络处理方法、装置、电子设备及存储介质 |
-
2021
- 2021-09-26 CN CN202111131388.5A patent/CN115878547A/zh active Pending
-
2022
- 2022-08-18 WO PCT/CN2022/113302 patent/WO2023045638A1/zh unknown
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107437110A (zh) * | 2017-07-11 | 2017-12-05 | 中国科学院自动化研究所 | 卷积神经网络的分块卷积优化方法及装置 |
US20200167405A1 (en) * | 2018-11-28 | 2020-05-28 | Electronics And Telecommunications Research Institute | Convolutional operation device with dimensional conversion |
CN109919311A (zh) * | 2019-03-13 | 2019-06-21 | 北京地平线机器人技术研发有限公司 | 生成指令序列的方法、执行神经网络运算的方法和装置 |
CN111738424A (zh) * | 2020-06-29 | 2020-10-02 | 湖南国科微电子股份有限公司 | 神经网络处理方法、装置、电子设备及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN115878547A (zh) | 2023-03-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2023045445A1 (zh) | 数据处理装置、数据处理方法及相关产品 | |
CN112633490B (zh) | 执行神经网络模型的数据处理装置、方法及相关产品 | |
CN112799599B (zh) | 一种数据存储方法、计算核、芯片和电子设备 | |
WO2023045446A1 (zh) | 计算装置、数据处理方法及相关产品 | |
CN112686379B (zh) | 集成电路装置、电子设备、板卡和计算方法 | |
CN112799598B (zh) | 一种数据处理方法、处理器及电子设备 | |
US11775808B2 (en) | Neural network computation device and method | |
WO2024149112A1 (zh) | 卷积算子的编译方法及相关产品 | |
CN112084023A (zh) | 数据并行处理的方法、电子设备及计算机可读存储介质 | |
WO2023045638A1 (zh) | 计算装置、利用计算装置实施卷积运算的方法及相关产品 | |
CN113469337B (zh) | 用于优化神经网络模型的编译方法及其相关产品 | |
CN112801276B (zh) | 数据处理方法、处理器及电子设备 | |
CN113850377A (zh) | 数据处理装置、数据处理方法及相关产品 | |
CN113850379A (zh) | 数据处理装置、数据处理方法及相关产品 | |
CN114692844A (zh) | 数据处理装置、数据处理方法及相关产品 | |
CN113867800A (zh) | 计算装置、集成电路芯片、板卡、电子设备和计算方法 | |
WO2022257980A1 (zh) | 计算装置、利用计算装置实施卷积运算的方法及相关产品 | |
WO2023087698A1 (zh) | 执行卷积运算的计算装置、方法及相关产品 | |
CN115878543A (zh) | 计算装置、利用计算装置实施卷积运算的方法及相关产品 | |
CN113469333B (zh) | 执行神经网络模型的人工智能处理器、方法及相关产品 | |
WO2023087814A1 (zh) | 计算装置、利用计算装置实施卷积运算的方法及相关产品 | |
WO2022135600A1 (zh) | 计算神经网络的装置、板卡、方法及可读存储介质 | |
CN115878541A (zh) | 计算装置、利用计算装置实施卷积运算的方法及相关产品 | |
CN113792867B (zh) | 运算电路、芯片和板卡 | |
CN115878542A (zh) | 计算装置、利用计算装置实施卷积运算的方法及相关产品 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22871696 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |