CN112540946A - Reconfigurable processor and method for calculating activation functions of various neural networks on reconfigurable processor - Google Patents
Reconfigurable processor and method for calculating activation functions of various neural networks on reconfigurable processor Download PDFInfo
- Publication number
- CN112540946A CN112540946A CN202011511272.XA CN202011511272A CN112540946A CN 112540946 A CN112540946 A CN 112540946A CN 202011511272 A CN202011511272 A CN 202011511272A CN 112540946 A CN112540946 A CN 112540946A
- Authority
- CN
- China
- Prior art keywords
- processing unit
- reconfigurable
- neural network
- processing units
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000006870 function Effects 0.000 title claims abstract description 233
- 230000004913 activation Effects 0.000 title claims abstract description 112
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012545 processing Methods 0.000 claims abstract description 317
- 238000004364 calculation method Methods 0.000 claims abstract description 58
- 230000002093 peripheral effect Effects 0.000 claims abstract description 21
- 238000003860 storage Methods 0.000 claims abstract description 21
- 230000005540 biological transmission Effects 0.000 claims abstract description 19
- 238000009825 accumulation Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 9
- 238000010586 diagram Methods 0.000 description 28
- 238000013461 design Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 7
- 230000002349 favourable effect Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 238000009795 derivation Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/167—Interprocessor communication using a common memory, e.g. mailbox
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/177—Initialisation or configuration control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/30003—Arrangements for executing specific machine instructions
- G06F9/30007—Arrangements for executing specific machine instructions to perform operations on data operands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline, look ahead
- G06F9/3867—Concurrent instruction execution, e.g. pipeline, look ahead using instruction pipelines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
Abstract
The embodiment of the invention provides a reconfigurable processor and a method for calculating a plurality of neural network activation functions on the reconfigurable processor, wherein the method comprises the following steps: splitting a neural network activation function into basic operations; according to the calculation sequence of each basic operation in the neural network activation function, input data are read from a shared memory through a reconfigurable processing array of a reconfigurable processor to sequentially realize each basic operation, processing units on the peripheral edge in the reconfigurable processing array can be used for executing access and storage operations and other operation operations, the processing units are called access and storage processing units, other processing units except the processing units on the peripheral edge in the reconfigurable processing array can be used for executing operation operations, the processing units on the peripheral edge and the processing units on the row or the column are used for executing operation operations, and each processing unit in the reconfigurable processing array and the processing units which are adjacent and exist in the upper, lower, left and right directions of the processing unit are used for data transmission.
Description
Technical Field
The invention relates to the technical field of reconfigurable processors, in particular to a reconfigurable processor and a method for calculating various neural network activation functions on the reconfigurable processor.
Background
In recent years, with the development of technologies such as artificial intelligence, cloud computing, big data and the like, the demand of human beings on computing is higher and higher, and the demand of human beings on chip performance is also higher and higher. However, as the chip size is gradually reduced and moore's law gradually approaches physical limits, the power of integrated circuits is difficult to continue to increase, and thus chip design is required to shift from an increase in power performance to an increase in energy efficiency and flexibility. Therefore, chip structure design in a dedicated field, which can be optimally designed for a certain field, becomes the mainstream of chip design today, and chip design with high performance, high energy efficiency ratio and high flexibility becomes an important index today.
Meanwhile, with the continuous development of the neural network, the network structure and the activation function are continuously changed, and for a special ASIC neural network accelerator, after the network structure and the activation function are changed, the acceleration effect is reduced to some extent, and even the special ASIC neural network accelerator is not suitable for a novel network any more.
Disclosure of Invention
The embodiment of the invention provides a method for calculating various neural network activation functions on a reconfigurable processor, which aims to solve the technical problem that an ASIC neural network accelerator in the prior art has low acceleration effect after the network structure and the activation functions are changed. The method comprises the following steps:
splitting a neural network activation function into basic operations;
according to the calculation sequence of each basic operation in the neural network activation function, input data are read from a shared memory through a reconfigurable processing array of a reconfigurable processor to sequentially realize each basic operation, wherein processing units on the peripheral edge in the reconfigurable processing array are used for executing access and storage operations and are called access and storage processing units, other processing units except the processing units on the peripheral edge in the reconfigurable processing array are used for executing operation operations and are called operation processing units, the processing units on the peripheral edge perform data transmission with the processing units on the row or the column for executing operation operations, and each processing unit in the reconfigurable processing array performs data transmission with the adjacent processing units existing in the upper, lower, left and right directions of the processing unit.
The embodiment of the invention also provides a reconfigurable processor for realizing the calculation of the activation functions of the various neural networks, so as to solve the technical problem that the acceleration effect of the ASIC neural network accelerator is low after the network structure and the activation functions are changed in the prior art. The reconfigurable processing array includes:
a shared memory for storing input data;
the reconfigurable processing array is used for reading input data from a shared memory to sequentially realize each basic operation according to the calculation sequence of each basic operation after the neural network activation function is split, wherein processing units on the peripheral edge in the reconfigurable processing array are used for executing access and storage operations and are called access and storage processing units, other processing units except the processing units on the peripheral edge in the reconfigurable processing array are used for executing operation operations and are called operation processing units, the processing units on the peripheral edge perform data transmission with the processing units on the row or the column for executing operation operations, and each processing unit in the reconfigurable processing array performs data transmission with the adjacent processing units existing in the upper, lower, left and right directions of the processing unit.
In the embodiment of the invention, the neural network activation function is divided into basic operations, and then the input data is read from the shared memory by the reconfigurable processing array according to the calculation sequence of each basic operation in the neural network activation function to sequentially realize each basic operation, so that the operation of the neural network activation function is realized on the existing reconfigurable processing array structure without changing the reconfigurable processing array structure or adding a circuit structure on the reconfigurable processing array structure, namely different processing units in the reconfigurable processing array are configured according to the algorithm requirements of different neural network activation functions to carry out corresponding operations, so that the complex activation function operation can be realized on the reconfigurable processing array structure by using basic operations such as addition, subtraction, multiplication, shift and the like, thereby being beneficial to simplifying the circuit design of the activation function operation, the method is favorable for improving the operation speed and the throughput rate of the circuit, and the operation algorithm of the processing units in the reconfigurable processing array can be flexibly configured and adopts the input and output mode of a production line, so that the method is favorable for meeting the operation of different changed activation functions, has expandability and is also favorable for improving the utilization rate of the processing units.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 is a flow chart of a method for calculating a plurality of neural network activation functions on a reconfigurable processor according to an embodiment of the present invention;
FIG. 2 is a graph illustrating a relu function according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating a calculation flow of a relu function according to an embodiment of the present invention;
fig. 4 is a schematic layout diagram of processing units in a reconfigurable processing array when a relu function is operated according to an embodiment of the present invention;
FIG. 5 is a graph illustrating a sigmoid function according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a calculation flow of a sigmoid function according to an embodiment of the present invention;
fig. 7 is a schematic layout diagram of processing units in a reconfigurable processing array when a sigmoid function is computed according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a segmentation function image in the case of computing a sigmoid function according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of accumulated slice function images during sigmoid function operation according to an embodiment of the present invention;
FIG. 10 is a graphical illustration of a tanh function provided by an embodiment of the present invention;
FIG. 11 is a schematic diagram illustrating a calculation flow of a tanh function according to an embodiment of the present invention;
fig. 12 is a schematic layout diagram of processing units in a reconfigurable processing array when computing a tanh function according to an embodiment of the present invention;
FIG. 13 is a schematic diagram of a computing flow of an anti-overflow process according to an embodiment of the present invention;
FIG. 14 is a schematic diagram illustrating an arrangement of processing units in a reconfigurable processing array during anti-overflow processing according to an embodiment of the present invention;
FIG. 15 shows a calculation e according to an embodiment of the present inventionxThe schematic diagram of the calculation flow of (1);
FIG. 16 shows a calculation e according to an embodiment of the present inventionxThe arrangement schematic diagram of the processing units in the time reconfigurable processing array;
FIG. 17 is a block diagram of a method for calculating ln (Σ e) according to an embodiment of the present inventionx) The schematic diagram of the calculation flow of (1);
FIG. 18 is a block diagram of a method for calculating ln (Σ e) according to an embodiment of the present inventionx) The arrangement schematic diagram of the processing units in the time reconfigurable processing array;
fig. 19 is a block diagram of a reconfigurable processor for implementing multiple neural network activation function computations according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
The inventor of the application finds that the coarse-grained reconfigurable processor architecture is receiving more and more attention due to the characteristics of low energy consumption, high performance, high energy efficiency and flexible and dynamic reconfigurability. The flexibility of the reconfigurable computing architecture is between that of a general-purpose processor and that of an ASIC processor, and meanwhile, the efficiency of the reconfigurable computing architecture can approach that of the ASIC processor through optimization, so that the reconfigurable computing architecture has the advantages of the general-purpose processor and the ASIC processor. Its characteristics determine that it is well suited for data intensive operations, which is in full agreement with the computational requirements of neural networks. In the calculation of the neural network, the implementation of the activation function is particularly important as a part for providing nonlinearity, however, unlike a dedicated ASIC processor, a coarse-grained reconfigurable processor does not have a circuit dedicated for processing the activation function, and if the activation function implementation circuit of the neural network is added to a reconfigurable calculation architecture, a certain redundancy is generated, and the complicated circuit design also causes the reduction of performance and the increase of power consumption. Therefore, the inventor of the present application proposes a plurality of methods for calculating neural network activation functions on the reconfigurable processor, so as to realize the operation of the more complicated neural network activation functions on the basis of the existing simpler design of the reconfigurable processing array circuit.
In an embodiment of the present invention, a method for calculating multiple neural network activation functions on a reconfigurable processor is provided, as shown in fig. 1, the method includes:
step 102: splitting a neural network activation function into basic operations;
step 104: according to the calculation sequence of each basic operation in the neural network activation function, input data are read from a shared memory through a reconfigurable processing array of a reconfigurable processor to sequentially realize each basic operation, wherein processing units on the peripheral edge in the reconfigurable processing array are used for executing access and storage operations and are called access and storage processing units, other processing units except the processing units on the peripheral edge in the reconfigurable processing array are used for executing operation operations and are called operation processing units, the processing units on the peripheral edge perform data transmission with the processing units on the row or the column for executing operation operations, and each processing unit in the reconfigurable processing array performs data transmission with the adjacent processing units existing in the upper, lower, left and right directions of the processing unit.
It can be known from the flow shown in fig. 1 that, in the embodiment of the present invention, it is proposed to split the neural network activation function into basic operations, and further sequentially implement each basic operation by reading input data from the shared memory through the reconfigurable processing array according to the calculation sequence of each basic operation in the neural network activation function, so as to implement the operation of the neural network activation function on the existing reconfigurable processing array structure, without changing the reconfigurable processing array structure, and without adding a circuit structure on the reconfigurable processing array structure, that is, different processing units in the reconfigurable processing array are configured to perform corresponding operations according to the algorithm requirements of different neural network activation functions, so that the complex activation function operations can be implemented on the reconfigurable processing array structure by using basic operations such as addition, subtraction, multiplication, and shift, thereby facilitating to simplify the circuit design of the activation function operations, the method is favorable for improving the operation speed and the throughput rate of the circuit, and the operation algorithm of the processing units in the reconfigurable processing array can be flexibly configured and adopts the input and output mode of a production line, so that the method is favorable for meeting the operation of different changed activation functions, has expandability and is also favorable for improving the utilization rate of the processing units.
In specific implementation, the operation of the neural network activation function can be divided into basic operations aiming at different neural network activation functions, and then the reconfigurable processing array reads input data from the shared memory to sequentially realize each basic operation. Specifically, for the same neural network activation function, the operation of the neural network activation function has expandability by adjusting the fineness of the split basic operation and different split schemes on the operation of the neural network activation function, and the requirements for different precisions and different throughput can be met. For example, under the requirement of low precision, the neural network activation function can be roughly split into fewer basic operations, so that the precision is reduced, and the throughput is improved; under the high-precision requirement, the neural network activation function can be finely divided into a plurality of basic operations, so that the precision is improved.
In a specific implementation, the basic operation may include: basic and simple operations such as addition, subtraction, multiplication, multiply-accumulate operation, shift operation, and selection operation. To enable the computation of complex neural network activation functions by performing simple basic operations on a reconfigurable processing array.
In particular implementations, for a linear, segmented neural network activation function, the operations may be performed on a reconfigurable processing array by, for example,
splitting a neural network activation function into basic operations, comprising:
for a linear segmented neural network activation function, splitting the neural network activation function into selection operations;
according to the calculation sequence of each basic operation in the neural network activation function, the basic operations are sequentially realized through a reconfigurable processing array, and the method comprises the following steps:
the method comprises the steps that input data are read from a shared memory through a plurality of access processing units in the reconfigurable processing array, the input data are transmitted to an operation processing unit of a row or a column where the access processing units are located through each access processing unit to carry out selection operation, calculation results of the selection operation are transmitted to the access processing unit of the row or the column where the access processing units are located through the operation processing units, and then calculation results are stored in the shared memory, wherein the access processing unit for reading the input data and the access processing unit for storing the calculation results are different access processing units, and the calculation results output by the different operation processing units are transmitted to the different access processing units.
In specific implementation, the linear segmented neural network activation function is exemplified by a linear rectification function (i.e., relu function), and the relu function is f (z) ═ max (0, x), as shown in fig. 2, the curve has characteristics of monotonous increase and easy derivation.
Specifically, how to map the hardware ASIC circuit algorithm implementation of the relu function to the architecture of the reconfigurable computing needs to be considered when the relu function is implemented on the architecture of the reconfigurable computing. Considering the ASIC circuit implementation principle of the relu function, it is necessary to take input data x out of the shared memory of the reconfigurable processing array, select the input data x by sel operation, determine whether the input data x is positive or negative, and select whether the final output is 0 or x.
In the following, the implementation of the relu function will be described by taking an example of a 4 × 4 reconfigurable processing array PEA (which is one quarter of the entire reconfigurable processing array, and the general reconfigurable processing array is 8 × 8). Firstly, as shown in table 1 below, we take input data from a shared memory by executing Load operation through a processing unit PE (i.e., the above-mentioned access processing unit) on the edge of a reconfigurable processing array, then realize sel operation through the processing unit PE (i.e., the above-mentioned operation processing unit) inside the reconfigurable processing array, select 0 or x for output, and finally, store the calculated result into a shared memory by executing Save operation through the processing unit PE on the edge of the reconfigurable processing array, specifically, the arrangement of different operations executed by each processing unit in the reconfigurable processing array is shown in fig. 4, wherein the access processing unit reading the input data and the access processing unit storing the calculated result are different access processing units to realize pipeline execution, the calculated results output by different operation processing units are transmitted to different access processing units, and then different access processing units can store the calculation results output by different operation processing units into the shared memory, so that data coverage is avoided.
TABLE 1
| Operation sign | Means of |
| Load | Fetching data in memory |
| Sel | Selecting, inputting a, b, c, selecting b or c to output according to the value of a |
| Save | Storing, storing data in memory |
In particular implementations, for neural network activation functions that are symmetric and allow a piecewise taylor expansion fit, the functions can be computed on a reconfigurable processing array by, for example,
splitting a neural network activation function into basic operations, comprising:
for a symmetrical neural network activation function allowing segmented Taylor expansion fitting, splitting the neural network activation function into a first symmetrical part and a second symmetrical part according to symmetry, dividing input data of the first symmetrical part into a plurality of data segments, sequentially splitting operation of each data segment into subtraction, selection operation and multiply-accumulate operation, adding the multiply-accumulate operation results of the data segments, subtracting the output maximum value of the first symmetrical part from the accumulated result, performing selection operation to obtain output data of the first symmetrical part, subtracting the output data of the first symmetrical part from the output maximum value of the first symmetrical part, and performing selection operation to obtain output data of the second symmetrical part;
according to the calculation sequence of each basic operation in the neural network activation function, the basic operations are sequentially realized through a reconfigurable processing array, and the method comprises the following steps:
sequentially subtracting a numerical value in each data segment from the shared memory through one memory access processing unit in the reconfigurable processing array, respectively subtracting the read numerical value from an end value of the divided data segment through a plurality of operation processing units, forming a first-stage selector through the plurality of operation processing units, wherein each operation processing unit in the first-stage selector corresponds to one data segment, and each operation processing unit in the first-stage selector outputs a minimum value in the read numerical value and a maximum value of the corresponding data segment based on a subtraction result; the second-stage selector is composed of a plurality of operation processing units, each operation processing unit in the second-stage selector corresponds to a previous data segment, a first operation processing unit in the second-stage selector outputs the output of the first operation processing unit in the first-stage selector, other operation processing units in the second-stage selector output the maximum value of the output of the corresponding operation processing unit in the first-stage selector and the maximum value of the previous data segment, the operation processing units respectively carry out multiply-accumulate operation on the outputs of the operation processing units in the second-stage selector, the operation processing units carry out addition operation on the results of the multiply-accumulate operation, the operation processing units subtract 1 from the result of the addition operation and carry out selection operation to obtain the output data of the first symmetric part, and the operation processing units subtract 1 from the output data of the first symmetric part and carry out selection operation, the output data of the second symmetric part is obtained.
In specific implementation, the above symmetric neural network activation functions allowing the piecewise taylor expansion fitting are exemplified by an S-shaped growth curve function (i.e., Sigmoid function) and a hyperbolic tangent function (i.e., Tanh function). Sigmoid function of
Is a common sigmoid function in biology. It can map the input variables between (0, 1), as shown in fig. 5, with the property of monotonic increase and easy derivation. In a neural network, if the output unit of the user deals with the binary problem, the sigmoid function can be obtained by using the generalized linear distribution, and the output result is the Bernoulli distribution.
In specific implementation, a lookup table can be hardly realized on a reconfigurable array based on a pipeline. Due to the change of the input data, the address of the access is changed. Based on a general reconfigurable array, the fetch address of a processing unit is generally realized by a base address and an offset address. If the reconfigurable array is used to implement a look-up table, the fetch address will change as the input data changes, causing the pipeline to stall. Therefore, the present embodiment proposes to perform piecewise integral accumulation on the function, and implement the calculation of the function in a pipelined manner. Specifically, the basic operation of splitting the Sigmoid function when the Sigmoid function is operated is shown in table 2 below.
TABLE 2
In specific implementation, firstly, according to the symmetry of sigmoid, we can know that only a part (namely the first symmetric part) with a function larger than 0 needs to be calculated, and finally, the other half of the function (namely the second symmetric part) can be obtained through rotation change. Therefore, we map all input data to the interval of [0, ∞).
Secondly, Taylor expansion is carried out on different parts of the sigmoid function to obtain an approximate function of the sigmoid function. Based on the reconfigurable processing array, the input data part of the sigmoid function at [0, + ∞) is divided into 4 data segments (in specific implementation, the number of data segments can be determined according to different precision requirements, and the higher the number of data segments is, the higher the precision is), which are [0, 4), [4, 8), [8, 15), [15, ∞) respectively.
Firstly, the data range of input data is judged by using a sel operation function. The input of the Sel operation function is a, b and c, and the Sel operation function can select any one of b or c to output according to the value of the input a. Firstly, subtraction is carried out on input data, and the input range is judged.
We construct a two-level selection function through the processing unit. The first-stage selection function is realized by three processing units, the smaller number of the two numbers is selected to be output, and the second-stage selection function is realized by three processing units, the larger number of the two numbers is selected to be output.
As shown in fig. 6 and 7, the range of the input data can be determined by subtracting 4, 8, and 15 (i.e., the end point values of the divided data segments) from the input data and by mapping the subtracted data. We will take 3 segments of input data as an example for analysis, and 1, 6, and 18 as examples for input data.
When the input data is 1, the first level of selectors is passed, wherein the first selector (whose inputs are 1 and 4) will output 1, the second selector (whose inputs are 1 and 8) will output 1, and the third selector (whose inputs are 1 and 15) will output 1. And then passing the output data of the first-stage selector through a second-stage selector, wherein the output of the first selector is 1, the output of the first selector in the first-stage selector is directly output through the first selector in the second-stage selector through routing operation, the output of the second selector (the input of the second selector is 1 and 4) is 4, and the output of the third selector (the input of the third selector is 1 and 8) is 8.
Similarly, when the input data is 6, the first selector will output 4, the second selector will output 6 and the third selector will output 6 when passing through the first stage of selectors. And the output data of the first-stage selector passes through a second-stage selector, the output of the first selector is 4, the output of the second selector (the inputs of the second selector are 6 and 4) is 6, and the output of the third selector (the inputs of the third selector are 6 and 8) is 8.
Similarly, when the input data is 18, the first selector will output 4, the second selector will output 8, and the third selector will output 15 when passing through the first stage of selectors. And the output data of the first-stage selector passes through a second-stage selector, the output of the first selector is 4, the output of the second selector (the inputs of the second selector are 8 and 4) is 8, and the output of the third selector (the inputs of the third selector are 8 and 18) is 18.
In summary, the sel operation function formed by the two-level selector can be expressed as the following formula (1)
sel(x,y,z)＝max(min(x,y),z)y＝4、5、8 z＝4、8 (1)
The output results of the three selectors of the second-stage selector are respectively processed by the processing units of three different paths through MAC operation, namely, Taylor expansion functions generated by expanding at three different points. And accumulating the signals to obtain a final output result. As shown in fig. 8, the function image of the solid line is a half sigmoid function, the function image of the "o" mark is a taylor function developed at [0, 4), "the function image of the" i "mark is a taylor function developed at [4, 8)," the function image of the "x" mark is a taylor function developed at [8, 15), "and the function image of the" x "mark is 1. By piecing them together, i.e. cumulatively, a new function can be obtained, as shown in fig. 9, it can be seen that the sigmoid function image is well fitted by the function of taylor expansion.
In the specific implementation, the segment intervals of the Sigmoid function are exemplified by [0, 4 ], [4, 8 ], [8, 15], [15, ∞) in consideration of the problem of accuracy loss. After (15, ∞), the result will be taken to be 1, with a loss of accuracy of about 10-7It can be ignored. In [0,15]]The interval is expanded to the third order by adopting a function expanded by a segmented Taylor function to obtain an approximate function, the specific precision loss and the Taylor expansion function are shown in the following table 3, and only the interval [0,30 ] is shown in the table]The negative interval can be obtained by the central symmetry about x ═ 0.
TABLE 3
| Taylor expansion function | Loss of maximum precision | |
| [0，4) | 3.56*10-3*x3-5.71*10-2*x2+2.93*10-1*x+4.92*10-1 | 7.53*10-3 |
| [4，8) | 4.96*10-4*x3-1.05*10-2*x2+7.51*10-2*x+8.19*10-1 | 5.23*10-4 |
| [8，15) | 3.21*10-6*x3-1.22*10-4*x2+1.54*10-3*x+9.94*10-1 | 3.71*10-5 |
| [15,∞) | 1 | 3.06*10-7 |
In specific implementation, when a PE on the reconfigurable processing array performs a normal operation, if the input of the PE is a and b, the output is the function f (a and b) executed by the PE, and a certain value of a and b can be selected as the output, specifically which value is output, depending on the position of the input a and b in the compiled instruction of the configuration PE, so that the specific operation and output performed by each PE in the reconfigurable processing array can be realized through configuration.
Specifically, the sigmoid function is calculated by adopting an implementation mode based on piecewise integral accumulation, and finally, the running water calculation of the function is realized based on the symmetry of the function, and can be realized by utilizing 3 global PEs and 28 processing unit PEs.
In particular, the Tanh function is
As shown in fig. 10, has characteristics of monotonous increase and easy derivation similar to the sigmoid function, while havingThe input variables can be mapped between (-1, 1).
Specifically, the operations of the Tanh function may be similar to the operations of the sigmoid function, except that the segmentation intervals are different, and [0,1 ], [1, 2 ], [2, 4 ], [4, ∞) are taken as examples. A calculation flowchart for calculating tanh is shown in fig. 11, a schematic layout of processing units in the reconfigurable processing array when the tanh function is calculated is shown in fig. 12, a specific precision loss and taylor expansion function are shown in table 4 below, where only the interval [0,15] is shown, and a negative interval can be obtained by central symmetry about x ═ 0.
TABLE 4
| Taylor expansion function | Loss of maximum precision | |
| [0，1) | 5.70*10-2*x3-4.57*10-1*x2+1.17*100*x-1.50*10-2 | 1.50*10-2 |
| [1，2) | 8.69*10-2*x3-0.559*10-1*x2+1.27*100* |
3.27*10-4 |
| [2，4) | 7.93*10-3*x3-8.42*10-2*x2+3.01*10-1*x+6.37*10-1 | 1.04*10-3 |
| [4,∞) | 1 | 6.71*10-4 |
In particular implementations, for neural network activation functions that include division, the operations on the reconfigurable processing array are implemented by, for example,
splitting a neural network activation function into basic operations, comprising:
for a neural network activation function comprising division, subtracting the maximum value of input data from the input data of the neural network activation function to avoid overflow, converting the division in the neural network activation function into subtraction, and dividing parameters participating in operation into different operation items according to the subtraction in the neural network activation function;
according to the calculation sequence of each basic operation in the neural network activation function, the basic operations are sequentially realized through a reconfigurable processing array, and the method comprises the following steps:
and sequentially realizing the operation of each operation item through the reconfigurable processing array.
In specific implementation, the neural network activation function including the division is, for example, Softmax, and the expression of Softmax is
By means of anti-overflow processing (i.e. replacing the input data x by x-x)max) The softmax function can be converted into
I.e. input the number x-xmaxTo thereby avoid exThe result of the function is too large resulting in overflow. Because the division is realized in a circuit more complexly, the invention adopts subtraction to replace the division, reduces the generated power consumption and the consumed resources, thereby improving the speed and the efficiency of the operation. By using logarithmic change, the softmax function can be converted into
Therefore, the operation on the softmax function is mainly divided into four parts, wherein the first part is an anti-overflow part, namely solving x-xmax(i.e., the above-described operation terms). The second part is the calculation of ex(i.e., the above-described operation terms). The third part is e to be obtainedxAdd up and find ln (Σ e)x) (i.e., the above-described operation terms). The fourth part is solving
(i.e., the above-described operation terms).
In specific implementation, in order to implement the anti-overflow processing, the maximum value of the input data is subtracted from the input data, and specifically, the maximum value of the input data is found out by, for example, dividing the input data into a plurality of data groups, for each data group, reading the input data through an access processing unit, receiving the input data through an operation processing unit and performing selective operation on the input data, outputting the maximum value of the data group, performing parallel processing on the plurality of data groups to obtain the maximum value of each data group, reading the maximum value of each data group through an access processing unit, receiving the maximum value of each data group through an operation processing unit and performing selective operation on the received data, and outputting the maximum value of the maximum values of each data group to obtain the maximum value of the input data.
Specifically, taking the operation item of the first step of calculating softmax as an example, the input data may be divided into 16 data groups, the operation for determining the maximum value in the input data may include the operation shown in table 5 below, the 16 data groups may be compared in parallel by the comparison operation of the processing array of the RPU, as shown in fig. 13 and 14, the access processing unit may perform load operation to read the input data of each data group from the shared memory, perform subtraction and selection operation by the operation processing unit to select the maximum value in each of the 16 data groups, and perform sel operation by the access processing unit to store the maximum value in each data group into the shared memory. Finally, the maximum values of the 16 data sets are compared with each other, thereby obtaining the maximum value of the input data. By utilizing the characteristic that the RPU can process data in parallel, the data processing is accelerated, and the efficiency is improved.
TABLE 5
| Operation sign | Means of |
| Load | Fetching data in memory |
| Sel | Selecting, inputting a, b, c, selecting b or c to output according to the value of a |
| - | Subtraction, input a, b, output a-b |
| Save | Storing, storing data in a memory |
In specific implementation, for an exponential function with e as the base in an operation item, this embodiment proposes that input data is read through a memory access processing unit, and then the input data is processed through an operation processing unitPerforming subtraction operation according to the maximum value of the input data, and performing subtraction operation on the subtraction result and the input data through an operation processing unit
Performing multiplication, converting an exponential function into an exponential function with the base 2, wherein the result of the multiplication is input data of the exponential function after bottom changing, the input data of the exponential function after bottom changing comprises an integer part and a decimal part, performing Taylor expansion on the exponential function with the base 2 and the decimal part as the exponent to obtain a polynomial, performing corresponding operation on the polynomial through an operation processing unit to obtain the output of the exponential function with the base 2 and the decimal part as the exponent, performing shift operation on the output and the integer part through the operation processing unit to obtain the output of the exponential function, and performing accumulation operation on the output of the exponential function through the operation processing unit.
Specifically, taking the operation term of the second step of calculating softmax as an example, e is calculatedx. It is noted that here the subtraction of x from the input data is takenmaxThereby preventing overflow. First, the bottom-changing formula, exBecome into
U in the formulaiFor the integer part, v, of the input data after variation using the bottom-changing formulaiThen the fractional part, yi＝x-xmax. Based on the characteristics of binary number, we can change the above formula again
At this time, the range of the input data is reduced to [ -1,0 [ ]]So that Taylor's expansion can be used to solve
We are right to
Performing Taylor expansion of
Finally, the obtained result is shifted to obtain
The result of (1).
In particular, calculating
As shown in fig. 15 and 16, the access processing unit is used to execute a load to execute a fetch operation, the input data is fetched from the memory, and x obtained by subtracting the overflow prevention processing of the previous stage from the subtraction is used to obtainmaxAnd finishing the anti-overflow and updating the data. Then, the data after the anti-overflow processing is multiplied with
Multiplication to obtain ui+viBy AND operation, u can be obtainediAnd viWill uiStoring, and performing multiply-accumulate by the arithmetic processing unitiPolynomial calculation is performed, and the specific calculation is formula (7). Finally, u is fetched from the memory by using the access operationiAnd shifting the result of the polynomial calculation to obtain the final output result, and storing it in a memory.
All exThrough addition and accumulation, the sigma e is obtainedxAnd stores it in memory for the next part of the calculation.
TABLE 6
| Operation sign | Means of |
| Load | Fetching data in memory |
| Sel | Selecting, inputting a, b, c, selecting b or c to output according to the value of a |
| And | And operation, input a, b, output a&b |
| >> | Shift operation, input a, output shifted a |
| + | Addition, input a, b, output a + b |
| - | Subtraction, input a, b, output a-b |
| * | Multiplication, input a, b, output a b |
| MAC | Multiply-accumulate, input a, b, c, perform ab + c |
| Save | Storing, storing data in a memory |
In specific implementation, for a logarithmic function with e as a base in an operation term, this embodiment proposes that an input term of the logarithmic function is accumulation of an exponential function with e as a base, the accumulation of the exponential function is converted into a product of an exponential function with w as an index with 2 as a base and k, a leading 0 operation is performed by an operation processing unit to obtain a value of w, the accumulation of the exponential function with e as a base is shifted to obtain a value of k, the logarithmic function is subjected to taylor expansion based on the value of w and the value of k to obtain a polynomial, and the polynomial is operated by the operation processing unit to obtain an output of the logarithmic function.
Specifically, taking the operation term of the third step of calculating softmax as an example, the obtained exAdd up and find ln (Σ e)x). The accumulation part can be synchronously realized in the operation item process of the second step of the operation of softmax, and the result is accumulated in the global register every time one result is calculated. While calculating ln (∑ e)x) The central idea of (1) is taylor function expansion. For ln (∑ e)x) By the following variations, can be obtained
ln(∑ex)＝ln(2w*k) (8)
According to exAs can be appreciated, Σ exMust be positive, so in a binary number, the number is stored in the original code. By shifting, we can get the value of k, and reduce the calculated data to [0,1]The interval enables calculation of taylor expansion. And the value of w is calculated by the leading 0. After obtaining the value of w, Σ exBy performing a shift operation, the value of k can be obtained. Then, the final calculation expression, equation (9), can be obtained by changing equation (8) and performing taylor expansion. Specifically, ln (∑ e) is calculatedx) The process of (a) calculates ln (Σ e) as shown in table 7 belowx) The calculation flow diagram of (1) is shown in FIG. 17, and ln (∑ e) is calculatedx) Fig. 18 is a schematic diagram showing the arrangement of processing units in the reconfigurable processing array.
TABLE 7
| Operation sign | Means of |
| Load | Fetching data in memory |
| Clz | Calculating leading 0, calculating the number of leading 0 in input data |
| + | Addition, input a, b, output a + b |
| - | Subtraction, input a, b, output a-b |
| * | Multiplication, input a, b, output a b |
| MAC | Multiply-accumulate, input a, b, c, perform ab + c |
| Save | Storing, storing data in a memory |
In specific implementation, in the operation item process of the fourth step of calculating softmax, solving is carried out
Since x has already been solved in the first stepmaxThe third step has already solved
The number to be subtracted is updated to
Bringing into the second step exAnd (4) calculating a function, wherein the calculation flow chart is completely the same as the calculation flow chart in the second step.
In the specific implementation process, in the process of sequentially realizing each basic operation through the reconfigurable processing array, when each operation processing unit needs to perform data transmission with the processing unit which is not in the row or the column, the processing unit which is in data transmission interconnection with the operation processing unit executes routing operation, so that the operation processing unit performs data transmission with the processing unit which is not in the row or the column; or, the data of the operation processing unit is output to a global register for storage, and the data is read by the processing unit of the operation processing unit which is not in the row or the column.
In specific implementation, simulation tests can be performed on the multiple neural network activation function calculation methods on the reconfigurable processor by adopting a python language, input data are random numbers between (-101, 101), the number of the input data is random numbers between (1, 100), and the turn is 100 times. According to the final simulation result, the maximum error is about 0.01, the maximum error is the precision of a 2-bit decimal number of 6-7 bits, the precision can be improved by improving the order of Taylor expansion, and the precision of the Taylor expansion is not improved in order to reduce power consumption and improve operation precision.
The calculation method of the neural network activation functions on the reconfigurable processor mainly realizes the calculation of the neural network activation functions on the reconfigurable architecture in a Taylor expansion mode. In the calculation of the softmax function, subtraction is adopted to replace division, and a bottom changing formula is combined with displacement to replace exIn a manner that reduces the number of coefficients and operations that need to be storedTime and thus further reduce the overhead of hardware resources of the device, thereby reducing area and power consumption.
In addition, the calculation method of the neural network activation functions on the reconfigurable processor has certain flexibility, and the expansion order can be determined for the application in a customized manner, so that the requirements of various precision data are met, and the power consumption, the calculation efficiency and the precision are well balanced.
Based on the same inventive concept, the embodiment of the present invention further provides a reconfigurable processor for implementing multiple neural network activation function computations, as described in the following embodiments. The reconfigurable processor for realizing the calculation of the multiple neural network activation functions has the advantages that the problem solving principle is similar to the problem solving principle of the calculation method of the multiple neural network activation functions on the reconfigurable processor, so the implementation of the reconfigurable processor for realizing the calculation of the multiple neural network activation functions can refer to the implementation of the calculation method of the multiple neural network activation functions on the reconfigurable processor, and repeated parts are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 19 is a block diagram of a reconfigurable processor for implementing multiple neural network activation function computations according to an embodiment of the present invention, as shown in fig. 19, including:
a shared memory 1902 for storing input data;
the reconfigurable processing array 1904 is configured to read input data from the shared memory to sequentially implement each basic operation according to a calculation sequence of each basic operation after splitting of a neural network activation function, where processing units on the peripheral edge in the reconfigurable processing array are used to perform access and storage operations and are referred to as access and storage processing units, other processing units in the reconfigurable processing array except the processing units on the peripheral edge are used to perform operation operations and are referred to as operation processing units, the processing units on the peripheral edge perform data transmission with the processing units on the row or the column where the processing units are located and used to perform operation operations, and each processing unit in the reconfigurable processing array performs data transmission with the processing units that are located and adjacent to each other in the vertical and horizontal directions of the processing unit.
In another embodiment, a software is provided, which is used to execute the technical solutions described in the above embodiments and preferred embodiments.
In another embodiment, a storage medium is provided, in which the software is stored, and the storage medium includes but is not limited to: optical disks, floppy disks, hard disks, erasable memory, etc.
The embodiment of the invention realizes the following technical effects: the method has the advantages that the neural network activation function is divided into basic operations, and then input data are read from a shared memory through a reconfigurable processing array according to the calculation sequence of the basic operations in the neural network activation function to sequentially realize the basic operations, so that the operation of the neural network activation function is realized on the existing reconfigurable processing array structure without changing the reconfigurable processing array structure or adding a circuit structure on the reconfigurable processing array structure, namely different processing units in the reconfigurable processing array are configured according to the algorithm requirements of different neural network activation functions to carry out corresponding operations, so that the complex activation function operation can be realized on the reconfigurable processing array structure by using the basic operations of addition, subtraction, multiplication, shift and the like, thereby being beneficial to simplifying the circuit design of the activation function operation and improving the operation speed and the throughput rate of the circuit, the operation algorithm of the processing units in the reconfigurable processing array can be flexibly configured and adopts a pipeline input and output mode, so that the operation of different changed activation functions is favorably met, the expandability is realized, and the utilization rate of the processing units is favorably improved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for calculating a plurality of neural network activation functions on a reconfigurable processor is characterized by comprising the following steps:
splitting a neural network activation function into basic operations;
according to the calculation sequence of each basic operation in the neural network activation function, input data are read from a shared memory through a reconfigurable processing array of a reconfigurable processor to sequentially realize each basic operation, wherein processing units on the peripheral edge in the reconfigurable processing array are used for executing access and storage operations and are called access and storage processing units, other processing units except the processing units on the peripheral edge in the reconfigurable processing array are used for executing operation operations and are called operation processing units, the processing units on the peripheral edge perform data transmission with the processing units on the row or the column for executing operation operations, and each processing unit in the reconfigurable processing array performs data transmission with the adjacent processing units existing in the upper, lower, left and right directions of the processing unit.
2. The method for multiple neural network activation function computation on a reconfigurable processor according to claim 1, wherein the basic operation includes: addition, subtraction, multiplication, multiply-accumulate operations, and selection operations.
3. The method for multiple neural network activation function computation on a reconfigurable processor according to claim 1,
splitting a neural network activation function into basic operations, comprising:
for a linear segmented neural network activation function, splitting the neural network activation function into selection operations;
according to the calculation sequence of each basic operation in the neural network activation function, the basic operations are sequentially realized through a reconfigurable processing array, and the method comprises the following steps:
the method comprises the steps that input data are read from a shared memory through a plurality of access processing units in the reconfigurable processing array, the input data are transmitted to an operation processing unit of a row or a column where the access processing units are located through each access processing unit to carry out selection operation, calculation results of the selection operation are transmitted to the access processing unit of the row or the column where the access processing units are located through the operation processing units, and then calculation results are stored in the shared memory, wherein the access processing unit for reading the input data and the access processing unit for storing the calculation results are different access processing units, and the calculation results output by the different operation processing units are transmitted to the different access processing units.
4. The method for multiple neural network activation function computation on a reconfigurable processor according to claim 1,
splitting a neural network activation function into basic operations, comprising:
for a symmetrical neural network activation function which can be fitted through segmented Taylor expansion, splitting the neural network activation function into a first symmetrical part and a second symmetrical part according to symmetry, dividing input data of the first symmetrical part into a plurality of data segments, sequentially splitting operation of each data segment into subtraction, selection operation and multiplication-accumulation operation, adding the multiplication-accumulation operation results of the data segments, comparing the accumulation result with the maximum output value of the first symmetrical part, performing selection operation to obtain output data of the first symmetrical part, subtracting the output data of the first symmetrical part from the maximum output value of the first symmetrical part, and performing selection operation to obtain output data of the second symmetrical part;
according to the calculation sequence of each basic operation in the neural network activation function, the basic operations are sequentially realized through a reconfigurable processing array, and the method comprises the following steps:
sequentially subtracting a numerical value in each data segment from the shared memory through one memory access processing unit in the reconfigurable processing array, respectively subtracting the read numerical value from an end value of the divided data segment through a plurality of operation processing units, forming a first-stage selector through the plurality of operation processing units, wherein each operation processing unit in the first-stage selector corresponds to one data segment, and each operation processing unit in the first-stage selector outputs a minimum value in the read numerical value and a maximum value of the corresponding data segment based on a subtraction result; a second selector is composed of a plurality of operation processing units, each operation processing unit in the second selector corresponds to a previous data segment, a first operation processing unit in the second selector outputs the output of the first operation processing unit in the first selector, other operation processing units in the second selector output the maximum value of the output of the corresponding operation processing unit and the maximum value of the previous data segment in the first selector, the operation processing units respectively multiply and accumulate the outputs of the operation processing units in the second selector, the operation processing units add the results of the multiply and accumulate operations, the operation processing units subtract the maximum value of the output of the first symmetric part from the result of the addition operation and select the result to obtain the output data of the first symmetric part, the operation processing units subtract the output data of the first symmetric part from the maximum value of the output of the first symmetric part and select the result, the output data of the second symmetric part is obtained.
5. The method for calculating the multiple neural network activation functions on the reconfigurable processor according to any one of claims 1 to 4,
splitting a neural network activation function into basic operations, comprising:
for a neural network activation function comprising exponential accumulation and exponential division, subtracting the maximum value of input data from the input data of the neural network activation function to prevent overflow, converting the division in the neural network activation function into subtraction, and dividing parameters participating in operation into different operation items according to the subtraction in the neural network activation function;
according to the calculation sequence of each basic operation in the neural network activation function, the basic operations are sequentially realized through a reconfigurable processing array, and the method comprises the following steps:
and sequentially realizing the operation of each operation item through the reconfigurable processing array.
6. The method for calculating the multiple neural network activation functions on the reconfigurable processor according to claim 5, wherein the operation of each operation item is sequentially realized through the reconfigurable processing array, and the method comprises the following steps:
dividing input data into a plurality of data groups, for each data group, reading the input data through an access processing unit, receiving the input data through an operation processing unit and carrying out selective operation on the input data, outputting the maximum value of the data group, carrying out parallel processing on the data groups to obtain the maximum value of each data group, reading the maximum value of each data group through an access processing unit, receiving the maximum value of each data group through an operation processing unit and carrying out selective operation on the received data, and outputting the maximum value of the maximum values of each data group to obtain the maximum value of the input data.
7. The method for calculating the multiple neural network activation functions on the reconfigurable processor according to claim 5, wherein the operation of each operation item is sequentially realized through the reconfigurable processing array, and the method comprises the following steps:
aiming at an exponential function with e as the base in an operation item, reading input data through a memory access processing unit, subtracting the maximum numerical value of the input data from the input data through an operation processing unit, and comparing the subtraction result with the maximum numerical value of the input data through the operation processing unit
Performing multiplication operation, converting an exponential function into an exponential function with the base 2, wherein the result of the multiplication operation is input data of the exponential function after the base is changed, the input data of the exponential function after the base is changed comprises an integer part and a decimal part, performing Taylor expansion on the exponential function with the base 2 and the decimal part as the exponent to obtain a polynomial, performing corresponding operation on the polynomial through an operation processing unit to obtain the output of the exponential function with the base 2 and the decimal part as the exponent, and performing operation on the polynomial through an operation positionThe processing unit performs shift operation on the output and the integer part to obtain the output of the exponential function, and the operation processing unit performs accumulation operation on the output of the exponential function.
8. The method for calculating the multiple neural network activation functions on the reconfigurable processor according to claim 7, wherein the operation of each operation item is sequentially realized through the reconfigurable processing array, and the method comprises the following steps:
for a logarithmic function with e as a base in an operation term, the input term of the logarithmic function is the accumulation of an exponential function with e as a base, the accumulation of the exponential function is converted into the product of an exponential function with w as an index with 2 as a base and k, leading 0 operation is carried out through an operation processing unit to obtain the value of w, the accumulation of the exponential function with e as a base is shifted to obtain the value of k, Taylor expansion is carried out on the logarithmic function based on the value of w and the value of k to obtain a polynomial, and the polynomial is operated through the operation processing unit to obtain the output of the logarithmic function.
9. The method for calculating the multiple neural network activation functions on the reconfigurable processor according to claim 5, wherein the step of sequentially implementing the basic operations through the reconfigurable processing array comprises:
in the process of sequentially realizing each basic operation through the reconfigurable processing array, when each operation processing unit needs to perform data transmission with a processing unit which is not in the row or not in the column, the processing unit which is interconnected with the operation processing unit in a data transmission manner executes routing operation, so that the operation processing unit performs data transmission with the processing unit which is not in the row or not in the column; or, the data of the operation processing unit is output to a global register for storage, and the data is read by the processing unit of the operation processing unit which is not in the row or the column.
10. A reconfigurable processor for implementing a plurality of neural network activation function computations, comprising:
a shared memory for storing input data;
the reconfigurable processing array is used for reading input data from a shared memory to sequentially realize each basic operation according to the calculation sequence of each basic operation after the neural network activation function is split, wherein processing units on the peripheral edge in the reconfigurable processing array are used for executing access and storage operations and are called access and storage processing units, other processing units except the processing units on the peripheral edge in the reconfigurable processing array are used for executing operation operations and are called operation processing units, the processing units on the peripheral edge perform data transmission with the processing units on the row or the column for executing operation operations, and each processing unit in the reconfigurable processing array performs data transmission with the adjacent processing units existing in the upper, lower, left and right directions of the processing unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011511272.XA CN112540946A (en) | 2020-12-18 | 2020-12-18 | Reconfigurable processor and method for calculating activation functions of various neural networks on reconfigurable processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011511272.XA CN112540946A (en) | 2020-12-18 | 2020-12-18 | Reconfigurable processor and method for calculating activation functions of various neural networks on reconfigurable processor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112540946A true CN112540946A (en) | 2021-03-23 |
Family
ID=75019265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011511272.XA Pending CN112540946A (en) | 2020-12-18 | 2020-12-18 | Reconfigurable processor and method for calculating activation functions of various neural networks on reconfigurable processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112540946A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115001955A (en) * | 2022-06-08 | 2022-09-02 | 苏州花园集信息科技有限公司 | Operation and maintenance data acquisition system and method thereof |
| WO2023116400A1 (en) * | 2021-12-20 | 2023-06-29 | 深圳市中兴微电子技术有限公司 | Vector operation method, vector operator, electronic device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106775599A (en) * | 2017-01-09 | 2017-05-31 | 南京工业大学 | Many computing unit coarseness reconfigurable systems and method of recurrent neural network |
| US20190042922A1 (en) * | 2018-06-29 | 2019-02-07 | Kamlesh Pillai | Deep neural network architecture using piecewise linear approximation |
| CN109409511A (en) * | 2018-09-25 | 2019-03-01 | 西安交通大学 | A kind of convolution algorithm data stream scheduling method for dynamic reconfigurable array |
| CN109472356A (en) * | 2018-12-29 | 2019-03-15 | 南京宁麒智能计算芯片研究院有限公司 | A kind of accelerator and method of restructural neural network algorithm |
| CN110516801A (en) * | 2019-08-05 | 2019-11-29 | 西安交通大学 | A kind of dynamic reconfigurable convolutional neural networks accelerator architecture of high-throughput |
| CN110688088A (en) * | 2019-09-30 | 2020-01-14 | 南京大学 | General nonlinear activation function computing device and method for neural network |
-
2020
- 2020-12-18 CN CN202011511272.XA patent/CN112540946A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106775599A (en) * | 2017-01-09 | 2017-05-31 | 南京工业大学 | Many computing unit coarseness reconfigurable systems and method of recurrent neural network |
| US20190042922A1 (en) * | 2018-06-29 | 2019-02-07 | Kamlesh Pillai | Deep neural network architecture using piecewise linear approximation |
| CN109409511A (en) * | 2018-09-25 | 2019-03-01 | 西安交通大学 | A kind of convolution algorithm data stream scheduling method for dynamic reconfigurable array |
| CN109472356A (en) * | 2018-12-29 | 2019-03-15 | 南京宁麒智能计算芯片研究院有限公司 | A kind of accelerator and method of restructural neural network algorithm |
| CN110516801A (en) * | 2019-08-05 | 2019-11-29 | 西安交通大学 | A kind of dynamic reconfigurable convolutional neural networks accelerator architecture of high-throughput |
| CN110688088A (en) * | 2019-09-30 | 2020-01-14 | 南京大学 | General nonlinear activation function computing device and method for neural network |
Non-Patent Citations (2)
| Title |
|---|
| 吕青;蒋林;邓军勇;李雪婷;: "面向对数与指数函数的可重构阵列结构", 微电子学与计算机, no. 10, 5 October 2016 (2016-10-05) * |
| 李昂;王沁;李占才;万勇;: "基于FPGA的神经网络硬件实现方法", 北京科技大学学报, no. 01, 25 January 2007 (2007-01-25) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023116400A1 (en) * | 2021-12-20 | 2023-06-29 | 深圳市中兴微电子技术有限公司 | Vector operation method, vector operator, electronic device and storage medium |
| CN115001955A (en) * | 2022-06-08 | 2022-09-02 | 苏州花园集信息科技有限公司 | Operation and maintenance data acquisition system and method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10379816B2 (en) | Data accumulation apparatus and method, and digital signal processing device | |
| CN110659015A (en) | Deep neural network architecture using piecewise linear approximation | |
| CN110163353B (en) | Computing device and method | |
| CN107305484B (en) | Nonlinear function operation device and method | |
| CN111915003A (en) | Neural network hardware accelerator | |
| CN112540946A (en) | Reconfigurable processor and method for calculating activation functions of various neural networks on reconfigurable processor | |
| EP3769208B1 (en) | Stochastic rounding logic | |
| CN113590195B (en) | Memory calculation integrated DRAM computing unit supporting floating point format multiply-add | |
| US20230221924A1 (en) | Apparatus and Method for Processing Floating-Point Numbers | |
| Ramachandran et al. | Performance analysis of mantissa multiplier and dadda tree multiplier and implementing with DSP architecture | |
| CN113902089A (en) | Device, method and storage medium for accelerating operation of activation function | |
| US20240126507A1 (en) | Apparatus and method for processing floating-point numbers | |
| US11551087B2 (en) | Information processor, information processing method, and storage medium | |
| Bruguera et al. | Design of a pipelined radix 4 CORDIC processor | |
| JP7354736B2 (en) | Information processing device, information processing method, information processing program | |
| US20200192633A1 (en) | Arithmetic processing device and method of controlling arithmetic processing device | |
| CN111178492B (en) | Computing device, related product and computing method for executing artificial neural network model | |
| WO2022126630A1 (en) | Reconfigurable processor and method for computing multiple neural network activation functions thereon | |
| JP7238376B2 (en) | Information processing system and information processing system control method | |
| Hsiao et al. | Design of a low-cost floating-point programmable vertex processor for mobile graphics applications based on hybrid number system | |
| CN109298848A (en) | The subduplicate circuit of double mode floating-point division | |
| CN116028011B (en) | Calculation method for random precision decimal data of GPU database | |
| Ueki et al. | Aqss: Accelerator of quantization neural networks with stochastic approach | |
| US20240111525A1 (en) | Multiplication hardware block with adaptive fidelity control system | |
| CN109416757A (en) | For handling the method, equipment and computer readable storage medium of numeric data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |