Disclosure of Invention
The invention aims to provide a parallel processing method based on a TOP-K segmentation hypercube model. The method takes a hypercube model parallel processing model as a research object, improves the parallel processing capability based on the hypercube model, and improves the processing time, the expandability and the speed-up ratio of the method compared with the original method.
The technical scheme is as follows:
a parallel processing method based on a TOP-K segmentation hypercube model,
wherein, k: a natural number k;
TOP-K: the kth large value;
o (n): the time complexity is n;
d: the number of recursions;
O(log2 2p): time complexity log2 2p;
n: the number and size;
p: the number of parallel processing units;
the method comprises the following steps:
step 1: finding the kth large value;
step 1.1: selecting a reference value;
step 1.2: taking the reference value selected in the step 1.1 as a demarcation point, placing the number smaller than the reference value on the left side, and placing the number larger than the reference value on the right side;
step 1.3: and judging the position of the reference value and the size of k, if the position of the reference value is equal to k, indicating that a median is found, if the position of the reference value is greater than k, recursing the algorithm on the left side in the step 2 to perform recursion operation, and if the position of the reference value is less than k, recursing the algorithm on the right side in the step 1.2 to perform recursion operation.
Step 2: the value obtained in step 1 is the TOP-K value of the problem, which is taken as the central element. And broadcasts the hub to all nodes. And dividing the data into two parts according to the central element, and randomly distributing the two parts on the sub hypercube nodes. The time complexity of this stage is O (n).
And step 3: and (5) after the
recursion step 1 and the
recursion step 2 are carried out for d times, the original data are distributed on the nodes of the early-child hypercube. The processor interconnection interchange data blocks interconnected along the communication link of the (d- (i-1)) th dimension are subjected to calculation processing. Since at most there are
The elements need to be sent between nodes since broadcasting the center element at the ith iteration requires O (d- (i-1)) time. Thus, in all d ═ log
2In p partitions, the time required for link communication is
And 4, step 4: and establishing a node interconnection link, and interconnecting and intercommunicating the nodes interconnected along the communication link of the nth dimension. The (n-1) -dimensional sub-hypercube with the nth bit of the node label being 0 contains the feature set in the original data, and the (n-1) -dimensional sub-hypercube with the nth bit of the node label being 1 contains the rest of the data in the original data. According to the programming goal, each node performs the calculation, and since p nodes are shared and the data is evenly distributed, the time complexity of the operation on the node is O ((n/p) log2p)。
And 5: each subcube recursively processes the owned subdata, and the data on each node are merged and summarized after being processed, so that a problem solving result is obtained. The combined solving time complexity of the segment with O (n/p) elements is log2(n/p), and finally, performing serial processing on each block of data, wherein the time complexity is O ((n/p) log2(n/p))。
Further, step 1.1 specifically comprises:
step 1.1.1: dividing n elements into
Each group comprises 5 elements, and if the rest elements are used as the filling number of 5 elements in the next division;
step 1.1.2: finding out the median of each group number in the step 1.1.1;
step 1.1.3: for all medians found in step 1.1.2 and the remainder in step 1.1.1, a new partition is made
Each group comprises 5 elements, and if the rest elements are used as the filling number of 5 elements in the next division;
step 1.1.4: and circularly calling the first three steps to obtain the median of the three steps as a reference value.
The invention has the beneficial effects that:
the parallel processing method based on the TOP-K segmentation hypercube model provided by the invention provides an optimal method based on the TOP-K segmentation, then improves the original parallel processing method based on the hypercube model according to the data segmentation mode, and further provides a parallel processing method based on the TOP-K segmentation hypercube model. The improved method has improved time complexity, expandability and speed-up ratio of the algorithm compared with the original method.
Detailed Description
The technical solution of the present invention will be described in further detail with reference to the following embodiments.
The effect of the invention is improved compared with the original method, which proves that:
suppose, in each of the d partitions, node P
1The size of the sub data stored in the medium is increased by k times, and 1 <
K 2. Thus, the total time spent in the d divisions is
When k > 1, the total time is O ((k)
d-1) n/p). Since p is 2
dThe above formula can be simplified into
When k is 2, then P1The time used for the division is O ((n-n/P), after d divisions, P1Has a sub-data size of 2dn/p。
When k is 1.1, the time taken for division is O ((p)0.138-1) n/p), the locally ordered sub-data size being n/p0.138。
When k is 1, the time taken for the division is O ((nlog)2p)/p), the sub-data size of the node is n/p, which is an ideal situation. It can be seen that the performance of the algorithm is worse when k is larger, as shown in table 1, and the trend of the total time change of the d-time division is shown in fig. 1.
Table 1: total time variation trend chart for d times of division
From the above trend of time-consuming total division, it is desirable that each time hypercube division is performed, sub-data of n/p size is allocated to each processing unit to be equally allocated, and such an allocation state is the best state. The total time taken for the segmentation is known from the previous derivation as O ((nlog)2p)/p) is also minimal.
Therefore, when the data is divided by taking the central element, if the selected central element is the median of the data segment, the data distributed to each processing unit is divided by hypercubeEvenly distributed, making the total time spent in the segmentation complex O ((nlog)2p)/p) is also minimal. The overall algorithm time complexity is reduced.
The invention provides a parallel processing method based on TOP-K segmentation hypercube model, the total algorithm execution time is
TpRefers to the total time a task runs on p nodes.
When (p log)2 2p)/(n*log2n)) ═ O (1), the solution is found to be p ═ O (n/log)2n)), therefore, when the number of nodes is equal to p, i.e., tasks are evenly distributed to each node, the distribution method is a cost-optimal strategy.
The method is proved to have improved parallel processing performance compared with the original hypercube.
Performance analysis and significant effects
The scalability scaleup and speed-up ratio speedup of parallel processing are two indexes for evaluating the quality of parallel processing, and scaleup refers to the performance of parallel processing after increasing the number of computers while enlarging the data size. scaleup (n), the time it takes to process a problem of size n using a single node running algorithm/the time it takes to process a problem of size n using n nodes running algorithms. If the scaleup value changes with n, if it is always around 1.0, or lower, it indicates that the algorithm has no effect on the change in size of the data set.
In the improved parallel processing method based on the TOP-K segmentation hypercube model, because
T when p approaches n
pIs composed of
scaleup(n)=(nT)/TpThe value can be reduced to
Hypercube model-based parallel processing method before improvement
The analysis follows from the method after improvement of the acceleration ratio, the algorithm acceleration ratio speed (n) is the time spent processing the problem using a single node running algorithm/the time spent processing the problem using n node running algorithms.
Therefore, the first and second electrodes are formed on the substrate,
when p approaches n, it can be simplified to
The index shows the trend of the time taken for the method to run as the number of nodes increases. If a linear increase can be maintained, it means that the time required for multi-node parallel processing can be shortened well.
According to the analysis of the time complexity, the expandability scaleup and the speed-up ratio speeddup of the parallel processing method based on the TOP-K segmentation hypercube model, the performance estimation table of the method can be obtained as shown in the table 2.
Table 2: performance estimation table
From table 2, the trend of the time complexity of the improved method can be seen, as shown in fig. 2 below.
It can be seen from fig. 4 that, as the scale n changes, the time complexity of the improved parallel processing method based on the TOP-K segmentation hypercube model is lower than that of the method before the improvement.
From table 2, the scalability scaleup trend of the improved parallel processing method based on the TOP-K segmentation hypercube model is shown in fig. 3. The scaleup value of the improved method changes from the initial lower 0.8 to the vicinity of 1.0 with the change of n, and shows the trend of changing from low to high. The representation method has good adaptability to both the size change of the data set and the size change of the processing node.
As can be seen from Table 2, the speed-up ratio speedup trend of the improved parallel processing method based on the TOP-K segmentation hypercube model is shown in FIG. 4.
As can be seen from FIG. 4, the speed value of the improved parallel processing method based on the TOP-K segmentation hypercube model changes with the data scale n, and the time change trend spent on running the algorithm can keep an approximately linear increase from small to large, which means that the method can well shorten the required time when processing data with large scale.
The above description is only a preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, and any simple modifications or equivalent substitutions of the technical solutions that can be obviously obtained by those skilled in the art within the technical scope of the present invention are within the scope of the present invention.