CN116957013A

CN116957013A - Floating point convolution operator acceleration method and device, electronic equipment and storage medium

Info

Publication number: CN116957013A
Application number: CN202310950528.4A
Authority: CN
Inventors: 张雪霏; 王晓雪
Original assignee: ARM Technology China Co Ltd
Current assignee: ARM Technology China Co Ltd
Priority date: 2023-07-28
Filing date: 2023-07-28
Publication date: 2023-10-27

Abstract

The embodiment of the application provides a floating point convolution operator acceleration method, a device, electronic equipment and a storage medium, and relates to the technical field of computers, wherein the method comprises the following steps: acquiring an image feature map to be processed, and carrying out target recognition on the image feature map to be processed through a target recognition network, wherein the target recognition network comprises: at least one convolution layer; the convolution processing of the image feature map to be processed through any convolution layer can specifically include: the method comprises the steps of obtaining a feature image to be processed and weight data and offset data corresponding to a convolution layer, wherein the feature image to be processed, the weight data and the offset data are floating point data, dividing the data into a symbol part, a base part and a mantissa part, aligning all the base parts, calling integer convolution operators to carry out convolution calculation, and carrying out regularization treatment on convolution calculation results, so that the time spent in the convolution calculation is reduced, and the network operation speed is improved.

Description

Floating point convolution operator acceleration method and device, electronic equipment and storage medium

Technical Field

The application belongs to the technical field of computers, and particularly relates to a floating point convolution operator acceleration method, a device, electronic equipment and a storage medium.

Background

The Convolution (Conv) layer is one of the most commonly used algorithm layers in the current network model, and almost all artificial intelligence and computer vision fields use this layer to develop applications of various industries, for example, the Conv layer can be applied to perform object recognition on images, so that it is important to optimize this layer.

In the related technology for carrying out target recognition on the image by applying the Conv layer, the floating point type Conv operator carries out convolution calculation on floating point type data to obtain a convolution calculation result so as to further realize target recognition, but when the floating point type Conv operator carries out convolution calculation on the floating point type data, the time is longer, and the network operation speed is slower.

Disclosure of Invention

The embodiment of the application aims to provide a floating point convolution operator acceleration method, a device, electronic equipment and a storage medium, which can reduce the time spent by convolution calculation and improve the running speed of a network. In order to achieve the above object, the technical solution provided by the embodiments of the present application is as follows:

In a first aspect, a floating-point convolution operator acceleration method is provided, including:

acquiring an image feature map to be processed, and carrying out target recognition on the image feature map to be processed through a target recognition network, wherein the target recognition network comprises: at least one convolution layer;

the convolution processing is carried out on the image feature map to be processed through any convolution layer, and the convolution processing comprises the following steps:

acquiring an image feature map to be processed and weight data and offset data corresponding to a convolution layer, wherein the feature map to be processed, the weight data and the offset data are floating point data;

the weight data and the offset data are respectively divided into a symbol part, a base part and a mantissa part for the image feature map to be processed;

aligning all the base parts, and calling integer convolution operators to carry out convolution calculation;

and regularizing the convolution calculation result.

In one possible implementation, the aligning the radix portions includes any one of:

fixing the value of each radix portion;

and performing a contrast process on the values of the base parts.

In another possible implementation, the fixing the values of the respective radix portions includes:

Respectively fixing the value of a first base part and the value of a second base part in a mode of shifting the mantissa part left, wherein the value of the first base part is the value of the base part corresponding to the image feature diagram to be processed, and the value of the second base part is the value of the base part corresponding to the weight data;

fixing a value of a corresponding third radix portion based on the value of the first radix portion and the value of the second radix portion, the value of the third radix portion being the value of the radix portion corresponding to the offset data;

wherein the value of the third radix portion is the sum of the value of the corresponding first radix portion and the value of the corresponding second radix portion.

In another possible implementation manner, the aligning the radix portions further includes:

determining the value of each corresponding mantissa portion based on the value of each aligned radix portion;

the step of calling integer convolution operators to carry out convolution calculation comprises the following steps:

and calling integer convolution operators to carry out convolution calculation based on the values of the mantissa parts corresponding to the respective mantissas.

In another possible implementation manner, the regularizing the convolution calculation result includes:

And shifting the convolution results of the corresponding mantissa parts left or right so that the shifted convolution calculation results meet the regularization requirement.

acquiring the value of a preset base part, and determining the value of a mantissa part corresponding to the value of the preset base part from the convolution result;

and regularizing the value of the corresponding mantissa part.

In another possible implementation manner, the invoking the integer convolution operator to perform convolution calculation further includes:

determining the space occupied by the image feature map to be processed and the maximum occupied space limited by the convolution calculation output result;

determining the maximum occupation space defined by the input of the integer convolution operator based on the space occupied by the image feature map to be processed and the maximum occupation space limited by the convolution calculation output result;

dividing the mantissa part into K parts, wherein K is an integer not less than 1, and K is determined based on the maximum space occupied by the mantissa part and the maximum occupied space defined by the convolution operator input of the integer;

and respectively calling integer convolution operators based on the value of the mantissa part corresponding to each part, and performing convolution calculation.

In a second aspect, there is provided a floating-point convolution operator acceleration apparatus comprising:

the acquisition module is used for acquiring an image feature map to be processed;

the target recognition module is used for carrying out target recognition on the image feature map to be processed through a target recognition network, wherein the target recognition network comprises: at least one convolution layer;

the target recognition module specifically includes: an acquisition unit, a first division unit, an alignment unit, a convolution calculation unit and a regularization processing unit, wherein,

the acquisition unit is used for acquiring the feature map of the image to be processed and weight data and offset data corresponding to the convolution layer, wherein the feature map to be processed, the weight data and the offset data are floating point data;

the first dividing unit is configured to divide the weight data and the offset data into a symbol portion, a radix portion, and a mantissa portion, respectively, for the image feature map to be processed;

The alignment unit is used for aligning all the cardinal parts;

the convolution calculation unit is used for calling integer convolution operators to carry out convolution calculation;

the regularization processing unit is used for regularizing the convolution calculation result.

In a possible implementation, the alignment unit is specifically configured to, when aligning the radix portions, any one of the following:

fixing the value of each radix portion;

and performing a contrast process on the values of the base parts.

In another possible implementation manner, the alignment unit is specifically configured to, when fixing the values of the respective radix portions:

In another possible implementation manner, the target identification module further includes: a first determination unit, wherein,

the first determining unit is used for determining the value of each corresponding mantissa part based on the value of each aligned base part;

the convolution calculation unit is specifically configured to, when invoking an integer convolution operator to perform convolution calculation:

In another possible implementation manner, the regularization processing unit is specifically configured to, when regularizing the convolution calculation result:

And regularizing the value of the corresponding mantissa part.

In another possible implementation manner, the target identification module further includes: a second determination unit, a third determination unit, and a second division unit, wherein,

the second determining unit is used for determining the space occupied by the image feature map to be processed and the maximum occupied space limited by the convolution calculation output result;

the third determining unit is configured to determine a maximum occupation space defined by the integer convolution operator input based on a space occupied by the image feature map to be processed and a maximum occupation space defined by the convolution calculation output result;

the second dividing unit is configured to divide the mantissa portion into K portions, where K is an integer not less than 1, and K is determined based on a maximum space occupied by the mantissa portion and a maximum occupied space defined by a convolution operator input of the integer;

In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, and the memory stores a computer program, and the processor executes the computer program to implement a floating-point convolution operator acceleration method provided in any one of possible implementations of the present application.

In a fourth aspect, embodiments of the present application also provide a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements a floating point convolution operator acceleration method provided in any one of the possible implementations of the present application.

In a fifth aspect, embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the floating point convolution operator acceleration method provided in any of the alternative embodiments of the present application.

The technical scheme provided by the embodiment of the application has the following beneficial effects:

the embodiment of the application provides a floating point convolution algorithm acceleration method, a device, electronic equipment and a storage medium, compared with the technology of carrying out target recognition on an image by applying a convolution layer, in the embodiment of the application, because a target recognition network comprises: the method comprises the steps of carrying out target recognition on an image feature map to be processed through a target recognition network, namely carrying out target recognition on the image feature map to be processed through the target recognition network, wherein when carrying out convolution processing through any one convolution layer, weight data and offset data corresponding to the image feature map to be processed and the convolution layer are all floating point type data, dividing the data into a symbol part, a base part and a mantissa part respectively, carrying out alignment processing on each base part, directly calling an integer type convolution operator to carry out calculation, and then carrying out regularization processing on a convolution calculation result to realize convolution calculation.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments of the present application will be briefly described below.

FIG. 1a is a schematic diagram of the Float16 data in an embodiment of the present application;

FIG. 1b is a schematic flow chart of a floating-point convolution operator acceleration method according to an embodiment of the present application;

FIG. 2 is a flow chart of another method for accelerating floating-point convolution operators in accordance with an embodiment of the present application;

FIG. 3a is a diagram illustrating mantissa alignment in an embodiment of the present application;

FIG. 3b is a schematic diagram of a mantissa portion for input range assurance in an embodiment of the present application;

FIG. 4 is a diagram illustrating an overall example of a floating-point convolution operator acceleration method in accordance with an embodiment of the present application;

FIG. 5a is a schematic diagram of a floating-point convolution operator acceleration apparatus according to an embodiment of the present application;

FIG. 5b is a schematic diagram of another floating-point convolution operator acceleration apparatus according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the drawings in the present application. It should be understood that the embodiments described below with reference to the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and the technical solutions of the embodiments of the present application are not limited.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and "comprising," when used in this specification, specify the presence of stated features, information, data, steps, operations, elements, and/or components, but do not preclude the presence or addition of other features, information, data, steps, operations, elements, components, and/or groups thereof, all of which may be included in the present specification. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein indicates that at least one of the items defined by the term, e.g., "a and/or B" may be implemented as "a", or as "B", or as "a and B". In describing a plurality of (two or more) items, if a relationship between the plurality of items is not explicitly defined, the plurality of items may refer to one, more or all of the plurality of items, for example, the description of "the parameter a includes A1, A2, A3" may be implemented such that the parameter a includes A1 or A2 or A3, and may also be implemented such that the parameter a includes at least two of three items of the parameters A1, A2, A3.

The embodiment of the application discloses a floating-point convolution operator acceleration method, which is based on the basic mathematical operation principle of floating points and skillfully converts floating-point Conv operation into int integer Conv operation, thereby optimizing the application of a floating-point Conv layer.

In the embodiment of the present application, a calculation formula of convolution is shown in formula 1, where:

wherein z represents conv calculation result, b represents conv offset, x represents conv input, w represents conv weight, k represents output channel coordinates, i, j represents result coordinates, u, v, k ' represents weight coordinates in a kernel, fs, fk represents kernel size, and i ', j ' represents input coordinates corresponding to a kernel weight; sh, sw represent the size of the stride in the h, w dimensions, respectively, and sh, sw represent the size of the kernel in the h, w dimensions, respectively.

As can be seen from equation 1, conv comprises summation and multiplication operations, specifically, multiplication followed by addition.

For the data of the Float16 type, the mathematical multiplication and addition operation mode of the Float16 type in the computer is as follows:

1. the flow 16 data is divided into a sign, radix (exponents) and mantissa (mantissas), as shown in fig. 1 a;

2. multiplication step 1: multiplying the symbol numbers, adding the base numbers, and multiplying the mantissas;

3. Multiplication step 2: regularizing the processing result to enable three parts of the sign, the exposure and the mantissa to meet the requirement of a flow 16;

4. addition step 1: base number contrast;

5. and an addition step 2: mantissa addition:

6. and an addition step 3: regularizing the result, such as 3.

As indicated above, intermediate step 3 can be omitted during the operation of multiplying and then adding the flow 16, as with the 3 and 6 effects. Furthermore, the cardinality can be permanently aligned by an operation given an exposure value, omitting step 4. Therefore, the above operation process can be simplified as:

1. the Float16 data is divided into sign, exposure and mantissa;

2. fixing the value of the exposure;

3. multiplication step 1: multiplying the number of symbols and the mantissa;

4. addition step 1: mantissa addition;

5. and regularizing the processing result.

Wherein, steps 3, and 4 are directly equivalent to integer conv operation. That is, by calling the integer type conv operator, the floating point type conv operation result can be calculated.

That is, the above method can be used to convert the convolution operation of the Float16 data by calling the floating point Conv operator into the convolution operation by calling the int integer Conv algorithm.

It should be noted that, any way of performing convolution operation on the flow 16 data by using the convolution layer may call the int integer convolution operator to perform convolution operation by using the method in the embodiment of the present application, and the scenario in the embodiment of the present application is not limited.

The technical solutions of the embodiments of the present application and technical effects produced by the technical solutions of the present application are described below by describing several exemplary embodiments. It should be noted that the following embodiments may be referred to, or combined with each other, and the description will not be repeated for the same terms, similar features, similar implementation steps, and the like in different embodiments.

Fig. 1b shows a flow chart of a floating-point convolution operator acceleration method according to an embodiment of the present application, where the method may be performed by any electronic device, for example, a user terminal (also referred to as a terminal, a terminal device, or a user device) or a server. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing service. The user terminal may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device (e.g., a smart speaker), a wearable electronic device (e.g., a smart watch), a vehicle-mounted terminal, a smart home appliance (e.g., a smart television), an AR/VR device, etc. The user terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment of the present application.

As shown in fig. 1b, the floating-point convolution operator acceleration method provided by the embodiment of the present application may include the following steps S101 to S102:

step S101, obtaining an image feature map to be processed.

In the embodiment of the application, the image feature image to be processed is obtained by extracting the image feature of the image to be processed. In the embodiment of the application, when the image feature extraction is carried out on the image to be processed, the convolution operation is carried out through the convolution layer, and the image feature extraction can be converted into the convolution calculation by calling the int integer convolution operator in the mode of the embodiment of the application.

Step S102, carrying out target recognition on the image feature map to be processed through a target recognition network.

Wherein the target recognition network comprises: at least one convolution layer. In the embodiment of the application, the image feature map to be processed is convolved at least through at least one convolution layer to perform target identification.

Specifically, the convolution processing of the image feature map to be processed through any convolution layer may specifically include: step Sa, step Sb, step Sc, and step Sd are shown in fig. 2, wherein,

step Sa, acquiring weight data and offset data corresponding to the image feature map to be processed and the convolution layer.

The feature map to be processed, the weight data and the offset data are floating point data. In the embodiment of the application, the feature map to be processed, the weight data and the offset data are all Float16 data.

And step Sb, dividing the image characteristic diagram to be processed, the weight data and the offset data into a symbol part, a base part and a mantissa part respectively.

Since the image feature map to be processed, the weight data and the offset data are all flow 16 data, after acquiring these data, each data is converted into three int types of data, namely sign, exp and mantissa.

And step Sc, aligning all the base parts, and calling integer convolution operators to carry out convolution calculation.

Specifically, aligning the respective radix portions may specifically include: the values of the respective radix portions are fixed or the values of the respective radix portions are subjected to a scalar processing. That is, in the embodiment of the present application, the radix may be always aligned by giving an exact value, or, alternatively, the radix may be aligned by performing radix-to-order processing.

After each radix part is aligned, convolution operation can be performed by calling an int integer convolution operator. In the embodiment of the application, after each radix part is aligned, the value of each corresponding mantissa part is determined based on each aligned radix part, and then the convolution operator of the int integer type is called to carry out convolution operation based on the determined value of each corresponding mantissa part.

And Sd, regularizing the convolution calculation result.

For the embodiment of the application, after convolution operation is performed by calling an int integer convolution operator, regularization processing is performed on the obtained convolution calculation result, so that the result (three parts of obtained sign, exponents and mantissa) after regularization processing accords with the rule of the Float 16.

Compared with the technology of performing target recognition on an image by applying a convolution layer, in the embodiment of the application, because the target recognition network comprises: the method comprises the steps of carrying out target recognition on an image feature map to be processed through a target recognition network, namely carrying out target recognition on the image feature map to be processed through the target recognition network, wherein when carrying out convolution processing through any one convolution layer, weight data and offset data corresponding to the image feature map to be processed and the convolution layer are all floating point type data, dividing the data into a symbol part, a base part and a mantissa part respectively, carrying out alignment processing on each base part, directly calling an integer type convolution operator to carry out calculation, and then carrying out regularization processing on a convolution calculation result to realize convolution calculation.

Further, as can be seen from the above embodiments, the values of the respective radix portions may be fixed so that the radix portions are aligned. In the embodiment of the present application, fixing the value of each radix portion may specifically include: fixing the value of the first base part and the value of the second base part respectively by shifting the mantissa part left; the value of the corresponding third radix portion is fixed based on the value of the first radix portion and the value of the second radix portion.

The values of the first base part are the values of the base parts corresponding to the image feature images to be processed, the values of the second base part are the values of the base parts corresponding to the weight data, and the values of the third base part are the values of the base parts corresponding to the offset data.

Notably, the radix alignment portion of the flow 16 addition operation is to large radix alignment because the mantissa right shift error for small radix is negligible compared to the radix alignment. However, in Conv operation, the large cardinality of different feature graphs has a large range of difference, but the fixed cardinality values are fixed for all feature graphs together, and if the large cardinality is selected to be aligned, the error is very large. Therefore, to ensure accuracy, only small radix alignments can be made. That is, in the embodiment of the present application, the value of the smallest radix portion is selected from the respective values (the value of the first radix portion and the value of the second radix portion) so that the value of the first radix portion and the value of the second radix portion are fixed by shifting the mantissa portion left based on the value of the smallest radix portion;

After the values of the first radix portion and the values of the second radix portion are fixed, the values of the third radix portion are the sum of the fixed values of the first radix portion and the second radix portion.

Further, in the embodiment of the present application, the alignment of the radix parts further includes: based on the values of the respective aligned radix portions, the values of the respective corresponding mantissa portions are determined. In an embodiment of the present application, after fixing the values of the first radix portion, the second radix portion, and the third radix portion, the values of the respective mantissa portions are determined based on the values of the fixed radix portions.

In the embodiment of the present application, after the values of the first radix portion and the second radix portion are fixed by a left shift manner, the values of the corresponding mantissa portions may be determined, and after the values of the third radix portion are determined by the values of the first radix portion and the values of the second radix portion, the values of the mantissa portions corresponding to the third radix portion (i.e., the values of the mantissas corresponding to the offset data) may be determined; the values of the first radix portion, the second radix portion, and the third radix portion may be fixed, and then the values of the mantissa portions may be determined.

And after aligning the values of all the radix parts and determining the values of the mantissa parts corresponding to the aligned values of all the radix parts, calling integer convolution operators, and calculating the convolution result of the mantissa. Specifically, invoking the integer convolution operator to perform convolution calculation may specifically include: and calling integer convolution operators to carry out convolution calculation based on the values of the mantissa parts corresponding to the respective values. That is, in the embodiment of the present application, the integer convolution operator is called to perform convolution calculation based on the value of the mantissa portion corresponding to the image feature map to be processed, the value of the mantissa portion corresponding to the weight data, and the value of the mantissa portion corresponding to the offset data, so as to obtain the convolution result.

Wherein the convolution result is still floating point type data.

It is noted that in actual operation, the code requirement cannot be satisfied because the ac operation intermediate result output of the called integer Conv operator has a limited range. Thus, invoking the integer convolution operator for convolution computation may also include: s1 (not shown), step S2 (not shown), and step S3 (not shown), wherein,

And S1, determining the space occupied by the image feature map to be processed and the maximum occupied space limited by the convolution calculation output result.

For example, the space occupied by the image Feature map to be processed, that is, the size of the Feature map is n bits (bits), and the maximum occupied space limited by the convolution calculation output result, that is, the output of the intermediate result of the Conv operator acc operation is limited to m bits.

And S2, determining the maximum occupied space defined by the input of the integer convolution operator based on the space occupied by the image feature map to be processed and the maximum occupied space defined by the convolution calculation output result.

For example, assuming that the input range is I bit, the acc intermediate result range is (2i+n) bit during Conv calculation, so that the Conv operator input (i.e., the maximum occupation space defined by the integer convolution operator input) is limited to i= (m bit-n bit)/2.

Further, in order to further secure the input range, step S3 may be performed.

And S3, dividing the mantissa part into K parts, wherein K is an integer not smaller than 1, and K is determined based on the maximum space occupied by the mantissa part and the maximum occupied space defined by the input of the convolution operator of the integer type.

The radix alignment step is known to align the input to a small radix, for example exp= -15 (5-bit base digit minimum), and the mantissa portion is at most 11bit+31bit (range: -15) =42 bit. To guarantee the input range, the mantissa portion is divided into k portions, k=42 bit/ibit+1, and when I belongs to the k range, each portion mantissa has a corresponding radix exp [ I ] = -15+i×l, and mantissa may be expressed as mantissa=Σmantissa [ I ] ×exp [ I ], where mantissa [ I ] is limited to the Ibit range, as shown in table 1. The initial mantissa is known as 11 bits, each bit corresponds to a radix of [ exp-10:exp ], and for mantissa [ i ], each bit corresponds to a radix of [ exp [ i ]: exp [ i ] +l-1], and no repetition occurs, as shown in Table 2:

TABLE 1

TABLE 2

Thus, each bit of the initial mantissa may be stuffed into a corresponding mantissa [ i ] of equal radix, thereby successfully dividing the initial mantissa into k groups of mantissa [ i ] in the l bit range.

In the above example, the output of the intermediate result of the Feature map= (3, 7), the common 8bit, and the Conv operator acc operation is limited to 32 bits, and the input value of the integer type Conv operator is limited to (32 bit-8 bit)/2=12 bit. The mantissa portion is known to be at most 42 bits, and to guarantee the input range, the mantissa is divided into 4 portions, exp= [15,5, -5, -15], with the mantissa being between adjacent radix ranges, respectively. For example, as shown in fig. 3a, when the radix exp=24-15=9, the 5 (11- (15-9) =5) bits before the mantissa corresponds to the radix between (27, 15), the 11 th bit of the most significant mantissa is omitted, thus showing that only the first 4 bits belong to the exp=15 portion, the 6 (15-9=6) bits after the mantissa corresponds to the radix between (15, 5), the former portion mantissa is aligned toward exp=15, and the latter portion mantissa is aligned toward exp=5.

It should be noted that in the above example, the initial mantissa has 11 bits, the most significant bit is omitted and 10 bits are left, each mantissa [ i ] that is sliced has 10 bits (except mantissa [0] has 12 bits), and is exactly divided into 2 parts, and aligned to exp=15 and exp=5 respectively, where the radix values corresponding to each bit in mantissa1 and mantissa2 are as shown in fig. 3b, and mantissa1 is equal to the first four bits of the original mantissa, mantissa2 is equal to the 6 bits after the original mantissa, and is shifted to the right by 4 bits.

In other different partitioning methods, u parts may be divided, and then, as shown in the above embodiment, the bit value of the base corresponding to the initial mantissa is only needed to be filled into the bits of the same base corresponding to the partitioned mantissa [ i ].

The mantissa corresponding radix meets the input requirement of the integer Conv operator. Step 3, (i.e. multiplication step 1: symbol number multiplication, mantissa multiplication) and step 4, (i.e. addition step 1: mantissa addition), 4×4=16 integer Conv operators are called, and finally mantissa calculation results are added and regularized. When the output of the intermediate result of the operation of the Conv operator acc is limited to 16 bits, the input of the Conv operator is limited to (16 bits-8 bits)/2=4bits, and mantissas are required to be divided into 10 parts, and the mantissa ranges are respectively between exp= [21, 17,3,9,5,1, -3, -7, -11, -15] adjacent radix ranges; when the output of the intermediate result of the operation of the Conv operator acc is limited to 48 bits, the input of the Conv operator is limited to (48 bits-8 bits)/2=20bits, and then mantissas are required to be divided into 2 parts, and the mantissa ranges are respectively between exp= [5, -15] adjacent radix ranges.

After obtaining the value of the mantissa part of each part in the K part, invoking an integer convolution operator to perform convolution calculation, which specifically comprises the following steps: and respectively calling integer convolution operators based on the value of the mantissa part corresponding to each part, and performing convolution calculation.

Further, regularizing the convolution calculation result may specifically include: the convolution results of the corresponding mantissa portions are shifted left or right, so that the shifted convolution calculation results meet the regularization requirement.

In the embodiment of the application, when the convolution calculation result is regularized, if the convolution result of the mantissa part is directly input, the result is beyond the 32bit integer type range. output_mantissa_tmp=Σmantissa [ i ] +2 (exp [ i ] +15), which is an input value directly taking mantissa as input, since exp= -15 (aligned to minimum radix) by default, output_mantissa_tmp is absolutely a value exceeding 32 bits. Therefore, to ensure the input value range, the regularization processing is performed on the convolution calculation result, which may specifically include: acquiring the value of a preset base part, and determining the value of a mantissa part corresponding to the value of the preset base part from the convolution result; regularizing the value of the corresponding mantissa part. That is, the possible base value is predetermined, and then the mantissa corresponding to the possible base value is used as the input, and the regularization processing is performed.

That is, the regularization processing is performed by the above embodiment, which ensures that the input value range satisfies the requirement and that errors are not generated as much as possible in the subsequent regularization processing. Specifically, for each mantisa [ i ], the bit value of the mantissa is found to be bitnum [ i ], and the mantissa corresponding base is known to be exp [ i ], so that the mantissa highest-order corresponding base value is known to be exp [ i ] +bitnum [ i ]. Taking exp=max (exp [ i ] +bitnum [ i ]) -15, the regularization result of each mantissa to radix exp is mantissa_tmp [ i ] =mantissa [ i ] < < (exp [ i ] -exp) or mantissa [ i ] > (exp-exp [ i ]), so as to ensure that the regularization result mantissa_tmp [ i ] is always in the 15bit range, so that the output_mantissa_tmp=Σmantissa_tmp [ i ] is added up by 6 numbers with large difference of 15 bits, and the result is always in the 11-19bit range. At this time, only the mantissa output_mantissa_tmp is required to be shifted to the right, so that the mantissa meets the regularization requirement, and the final output_mantissa is 11 bits.

In the following embodiment, taking the example that the output limit of the intermediate result of the Conv operator acc operation is 48bit,Feature map = (3, 7) and 8 bits altogether as an example, the acceleration mode of the floating-point convolution algorithm shown in the embodiment of the present application is described, as shown in fig. 4:

Acquiring an image Feature map to be processed (i.e., feature map described above), weight data and offset data, wherein the data are all flow 16 type data, dividing the data into Sign, exp and Mantissa respectively, for example, dividing Feature map= (3, 7) to obtain: sign=input [0], exp=input [1:6], mantissa=input [6: ]; aiming at the image Feature graph to be processed and the weight data, an aligned radix part is obtained, and the description is given by taking Feature as an example, and the input limit of int conv is as follows: (48-8)/2=20bit, so that the cardinality of Feature and weight can be fixed to [5, -15], the fixed cardinality of bias is [10, -10, -30], the value of the Mantissa portion is determined based on the fixed cardinality, i.e., based on mantissa0=mantissa > > { input_exp [0] -exp } and mantissa1=mantissa < < { exp-input_exp [1] }, resulting in input_exp= [5, -15], for offset data, mantissa0=mantissa > > { output_exp [0] -exp },

Mantissa1＝mantissa<<{exp-output_exp[1]}，

mantissa 2=mantissa < < { exp-output_exp [2] } to obtain out_exp= [10, -10, -30], and then determining the value of each corresponding Mantissa part, i.e., mantissa, from the aligned radix parts, and then performing convolution calculation based on the Mantissa and the following formula:

Out_mantissa0＝conv{input_mantissa0,weight_mantissa0,biases_mantiss a0}；

Out_mantissa1＝conv{input_mantissa0,weight_mantissa1,biases_mantiss a1}+conv{input_mantissa1,weight_mantissa0}；

Out_mantissa2＝conv{input_mantissa1,weight_mantissa1,biases_mantiss a2}；

After performing convolution calculation, obtaining a convolution calculation result, namely out_mantissa, further performing regularization treatment, returning a final output result, and outputting, wherein the output result is also data of the type of Float 16: exp [ I ] = bitnum (output_mantissa [ I ])+output_exp [ I ];

Exp＝max(exp[i])-15；

Mantissa+＝output_mantissa[I]>>{exp-output_exp[I]}；

Sign＝1,Mantissa＝-mantissa<0；

Mantissa<-mantissa right k shift to 11bit,Exp+＝k；

Output＝{{sign<<15)|{exp<<10}|{mantissa&0x3ff}}。

further, determining the possible range of the base number, when the mantissa is 15 bits (slightly more than 11 bits, and after the integers of the range of 6 bits and 15 bits are added or subtracted, the result is still slightly more than 11 bits and less than 32 bits), the corresponding base number sizes are determined, and the maximum value of the base number is the initial base number. It is known that the mantissa corresponding to the base must be slightly larger than 11 bits and smaller than 32 bits. Therefore, only the mantissa is required to be subjected to right shift processing, so that the mantissa can meet the regularization requirement. Thus, regularization processing efficiency can be greatly improved.

By the floating-point convolution data acceleration method, the running efficiency of a floating-point Conv layer on some platforms can be obviously improved. Taking input shape as [1,224,224,3] and the storage range of the acc intermediate result as 32 bits as an example, on a platform with a 1GHz main frequency, the running time of the original Float 16-type Conv layer is 90ms, the running time of the optimized Float 16-type Conv layer is expected to be reduced to 5.216ms, the efficiency is improved by 17 times, and the effect is obviously improved.

In addition, in a simulation experiment, an int integer convolution operator is called to carry out convolution calculation on floating point data in a mode shown in the embodiment of the application, and compared with direct conv operation on floating point data, the maximum binary error is 1, and the maximum relative error is 0.09 percent and is within an expected error range.

It should be noted that, the method for optimizing the calculation of floating point multiplication and addition and subtraction in Conv, which is shown in the embodiment of the present application, namely, the idea of quasi-converting floating point mathematical operation to integer mathematical operation, can be generalized to any process with the result of requiring floating point calculation. The manner in which the embodiments of the application are illustrated is not intended to be limiting.

Based on the same principle as the floating-point convolution algorithm acceleration method provided by the embodiment of the present application, the embodiment of the present application further provides a floating-point convolution algorithm acceleration device, which may be implemented as any electronic device, as shown in fig. 5a, where the floating-point convolution algorithm acceleration device 50 may include an acquisition module 51 and a target recognition module 52. Wherein:

an acquiring module 51, configured to acquire an image feature map to be processed;

the object recognition module 52 is configured to perform object recognition on the image feature map to be processed through an object recognition network, where the object recognition network includes: at least one convolution layer;

As shown in fig. 5b, when the target recognition module 52 performs convolution processing on the image feature map to be processed through any convolution layer, the method specifically includes: an acquisition unit 521, a first division unit 522, an alignment unit 523, a convolution calculation unit 524, and a regularization processing unit 525, wherein,

the obtaining unit 521 is configured to obtain the feature map of the image to be processed and weight data and offset data corresponding to the convolution layer, where the feature map, the weight data and the offset data to be processed are floating point data;

a first dividing unit 522, configured to divide the image feature map to be processed, the weight data and the offset data into a symbol portion, a radix portion and a mantissa portion, respectively;

an alignment unit 523 for aligning the respective radix portions;

the convolution calculation unit 524 is used for calling integer convolution operators to perform convolution calculation;

the regularization processing unit 525 is configured to perform regularization processing on the convolution calculation result.

Specifically, the alignment unit 523 is specifically used for any of the following when aligning the respective radix portions:

fixing the value of each radix part;

the values of the respective radix portions are subjected to a scalar processing.

Specifically, the alignment unit 523 is specifically configured to, when fixing the values of the respective radix portions:

The method comprises the steps of respectively fixing the value of a first base part and the value of a second base part in a mode of shifting a mantissa part left, wherein the value of the first base part is the value of the base part corresponding to the image feature graph to be processed, and the value of the second base part is the value of the base part corresponding to the weight data;

fixing a value of a corresponding third radix portion based on the value of the first radix portion and the value of the second radix portion, the value of the third radix portion being a value of the radix portion to which the offset data corresponds;

In one possible implementation of an embodiment of the present application, the object recognition module 52 further includes: a first determination unit, wherein,

a first determining unit configured to determine values of respective corresponding mantissa portions based on values of the respective aligned radix portions;

the convolution calculation unit 524 is specifically configured to, when invoking a convolution operator of an integer type to perform convolution calculation:

and calling integer convolution operators to carry out convolution calculation based on the values of the mantissa parts corresponding to the respective values.

Specifically, the regularization processing unit 525 is specifically configured to, when performing regularization processing on the convolution calculation result:

The convolution results of the corresponding mantissa portions are shifted left or right, so that the shifted convolution calculation results meet the regularization requirement.

regularizing the value of the corresponding mantissa part.

Another possible implementation of an embodiment of the present application, the object recognition module 52 further includes: a second determination unit, a third determination unit, and a second division unit, wherein,

the third determining unit is used for determining the maximum occupied space limited by the input of the integer convolution operator based on the space occupied by the image feature map to be processed and the maximum occupied space limited by the convolution calculation output result;

the second dividing unit is used for dividing the mantissa part into K parts, wherein K is an integer not less than 1, and K is determined based on the maximum space occupied by the mantissa part and the maximum occupied space defined by the input of the convolution operator of the integer type;

Note that the first dividing unit 522 and the second dividing unit may be the same dividing unit or may not be the same dividing unit; the first determining unit, the second determining unit, and the third determining unit may be the same determining unit, may not be the same determining unit, may be partially the same determining unit, and are not limited in the embodiment of the present application.

Compared with the technology of performing target recognition on an image by using a convolution layer, in the embodiment of the application, because the target recognition network comprises: the method comprises the steps of carrying out target recognition on an image feature map to be processed through a target recognition network, namely carrying out target recognition on the image feature map to be processed through the target recognition network, wherein when carrying out convolution processing through any one convolution layer, weight data and offset data corresponding to the image feature map to be processed and the convolution layer are all floating point type data, dividing the data into a symbol part, a base part and a mantissa part respectively, carrying out alignment processing on each base part, directly calling an integer type convolution operator to carry out calculation, and then carrying out regularization processing on a convolution calculation result to realize convolution calculation.

The device of the embodiment of the present application may execute the method provided by the embodiment of the present application, and its implementation principle is similar, and actions executed by each module and each unit in the device of the embodiment of the present application correspond to steps in the method of the embodiment of the present application, and detailed functional descriptions of each module of the device may be referred to in the corresponding method descriptions shown in the foregoing, which are not repeated herein.

An embodiment of the present application further provides an electronic device, where the electronic device includes at least one processor configured to perform the steps of the method provided in any of the alternative embodiments of the present application, and optionally, the electronic device may further include a transceiver and/or a memory coupled to the processor, where the memory stores a computer program, and where the processor may implement the solution provided in any of the alternative embodiments of the present application when the computer program is executed. Alternatively, the electronic device may be a user terminal or a server.

Fig. 6 shows a schematic structural diagram of an electronic device, which may be a server or a user terminal, and may be used to implement the method provided in any of the embodiments of the present application, as shown in fig. 6, where the embodiment of the present application is applicable.

As shown in fig. 6, the electronic device 600 may mainly include at least one processor 601 (one is shown in fig. 6), a memory 602, a communication module 603, and an input/output interface 604, and optionally, the components may be connected to each other by a bus 605. It should be noted that the structure of the electronic device 600 shown in fig. 6 is only schematic, and does not limit the electronic device to which the method provided in the embodiment of the present application is applicable.

The memory 602 may be used to store an operating system, application programs, and the like, and the application programs may include a computer program that implements the methods of the embodiments of the present application when called by the processor 601, and may include programs for implementing other functions or services. The Memory 602 may be, but is not limited to, a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory ) or other type of dynamic storage device that can store information and computer programs, an EEPROM (Electrically Erasable Programmable Read Only Memory ), a CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

The processor 601 is connected to the memory 602 via the bus 605 and realizes the corresponding functions by calling the application programs stored in the memory 602. The processor 601 may be, among other things, a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof that can implement or perform the various exemplary logic blocks, modules and circuits described in connection with the present disclosure. The processor 601 may also be a combination that performs computing functions, such as including one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

The electronic device 600 may connect to a network through a communication module 603 (which may include, but is not limited to, a component such as a network interface) to enable interaction of data, such as sending data to or receiving data from other devices (e.g., user terminals or servers, etc.) through communication of the network with the other devices. Wherein the communication module 603 may comprise a wired network interface and/or a wireless network interface, etc., i.e. the communication module may comprise at least one of a wired communication module or a wireless communication module.

The electronic device 600 may be connected to a required input/output device, such as a keyboard, a display device, etc., through the input/output interface 604, and the electronic device 600 may itself have a display device, or may be externally connected to other display devices through the interface 604. Optionally, a storage device, such as a hard disk, may be connected through the interface 604, so that data in the electronic device 600 may be stored in the storage device, or data in the storage device may be read, and data in the storage device may be stored in the memory 602. It is understood that the input/output interface 604 may be a wired interface or a wireless interface. The device connected to the input/output interface 604 may be a component of the electronic device 600, or may be an external device connected to the electronic device 600 when needed, according to the actual application scenario.

The bus 605 used to connect the various components may include a path to transfer information between the components. Bus 605 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect Standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The bus 605 may be classified into an address bus, a data bus, a control bus, and the like according to functions.

Alternatively, for the solution provided in the embodiment of the present application, the memory 602 may be used to store a computer program for executing the solution of the present application, and the processor 601 executes the computer program, where the processor 601 executes the computer program to implement the actions of the method or the apparatus provided in the embodiment of the present application.

Based on the same principle as the method provided by the embodiment of the present application, the embodiment of the present application provides a computer readable storage medium, where a computer program is stored, where the computer program can implement the corresponding content of the foregoing method embodiment when executed by a processor. Compared with the technology of performing object recognition on the image by applying the convolution layer, in the embodiment of the application, the object recognition network comprises: the method comprises the steps of carrying out target recognition on an image feature map to be processed through a target recognition network, namely carrying out target recognition on the image feature map to be processed through the target recognition network, wherein when carrying out convolution processing through any one convolution layer, weight data and offset data corresponding to the image feature map to be processed and the convolution layer are all floating point type data, dividing the data into a symbol part, a base part and a mantissa part respectively, carrying out alignment processing on each base part, directly calling an integer type convolution operator to carry out calculation, and then carrying out regularization processing on a convolution calculation result to realize convolution calculation.

Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the respective aspects of the method embodiments described above. Compared with the technology of performing object recognition on the image by applying the convolution layer, in the embodiment of the application, the object recognition network comprises: the method comprises the steps of carrying out target recognition on an image feature map to be processed through a target recognition network, namely carrying out target recognition on the image feature map to be processed through the target recognition network, wherein when carrying out convolution processing through any one convolution layer, weight data and offset data corresponding to the image feature map to be processed and the convolution layer are all floating point type data, dividing the data into a symbol part, a base part and a mantissa part respectively, carrying out alignment processing on each base part, directly calling an integer type convolution operator to carry out calculation, and then carrying out regularization processing on a convolution calculation result to realize convolution calculation.

It should be noted that the terms "first," "second," "third," "fourth," "1," "2," and the like in the description and claims of the present application and in the above figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate, such that the embodiments of the application described herein may be implemented in other sequences than those illustrated or otherwise described.

It should be understood that, although various operation steps are indicated by arrows in the flowcharts of the embodiments of the present application, the order in which these steps are implemented is not limited to the order indicated by the arrows. In some implementations of embodiments of the application, the implementation steps in the flowcharts may be performed in other orders as desired, unless explicitly stated herein. Furthermore, some or all of the steps in the flowcharts may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or phases may be performed at the same time, or each of these sub-steps or phases may be performed at different times, respectively. In the case of different execution time, the execution sequence of the sub-steps or stages can be flexibly configured according to the requirement, which is not limited by the embodiment of the present application.

The foregoing is merely an optional implementation manner of some of the implementation scenarios of the present application, and it should be noted that, for those skilled in the art, other similar implementation manners based on the technical ideas of the present application are adopted without departing from the technical ideas of the scheme of the present application, and the implementation manner is also within the protection scope of the embodiments of the present application.

Claims

1. A method for accelerating a floating-point convolution operator, comprising:

And regularizing the convolution calculation result.

2. The method of claim 1, wherein the aligning of the respective radix portions comprises any of:

fixing the value of each radix portion;

and performing a contrast process on the values of the base parts.

3. The method of claim 2, wherein said fixing the value of each radix portion comprises:

4. A method according to any one of claims 1-3, wherein said aligning the respective radix portions further comprises, after that:

5. The method of claim 4, wherein regularizing the convolution results comprises:

6. The method of claim 4, wherein regularizing the convolution results comprises:

and regularizing the value of the corresponding mantissa part.

7. The method of claim 1, wherein invoking an integer convolution operator for convolution computation further comprises:

8. A floating point convolution operator acceleration apparatus, comprising:

the alignment unit is used for aligning all the cardinal parts;

9. An electronic device comprising a memory in which a computer program is stored and a processor that, when running the computer program, performs the floating point convolution operator acceleration method of any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that the storage medium has stored therein a computer program which, when executed by a processor, implements the floating point convolution operator acceleration method of any one of claims 1 to 7.