CN108491187B - Parallel processing method based on TOP-K segmentation hypercube model - Google Patents
Parallel processing method based on TOP-K segmentation hypercube model Download PDFInfo
- Publication number
- CN108491187B CN108491187B CN201810156673.4A CN201810156673A CN108491187B CN 108491187 B CN108491187 B CN 108491187B CN 201810156673 A CN201810156673 A CN 201810156673A CN 108491187 B CN108491187 B CN 108491187B
- Authority
- CN
- China
- Prior art keywords
- data
- reference value
- hypercube
- elements
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 230000011218 segmentation Effects 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 34
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000004891 communication Methods 0.000 claims description 8
- 238000005192 partition Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000002123 temporal effect Effects 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 abstract description 15
- 230000001133 acceleration Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/22—Procedural
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The invention discloses a parallel processing method based on a TOP-K segmentation hypercube model, belongs to the technical field of data processing, and aims at the problems in the prior art. The improved method has improved time complexity, expandability and speed-up ratio of the algorithm compared with the original method.
Description
Technical Field
The invention belongs to the technical field of data processing, and relates to a parallel processing method based on a TOP-K segmentation hypercube model.
Background
As the amount of computing data varies, data processing also goes through parallel computing, distributed processing, and cloud computing processing methods. Parallel processing is an effective method for data processing, and can utilize a plurality of processing units and a storage unit to process data simultaneously to realize quick solution of problems. In the parallel processing process, each processing unit has a storage unit and a communication mode of interconnection and intercommunication among the processing units. If the processing object is larger data, the parallel processing method firstly divides the data set and then maps the data subset to each processing unit. The parallel execution of the control program is also considered while the parallel computation of the data objects is considered, so the time complexity of the parallel processing method must consider not only the complexity of the data computation but also the data communication complexity.
In a hypercube model based parallel processing method, the hypercube is segmented to form segmented data blocks if the first selected center element is exactly the smallest or largest element. Then, after the first segmentation, all elements will be concentrated onto one (n-1) -dimensional sub-hypercube, while the other (n-1) -dimensional sub-hypercube is empty. In subsequent operations, only half of the nodes continue to operate at most, while the other half is idle. This wastes the split time and increases the data processing time. Therefore, the size of each segmented sub-array data will affect the time complexity of the hypercube parallel processing algorithm.
In the prior art, a parallel processing method based on a TOP-K segmentation hypercube model is urgently needed.
Disclosure of Invention
The invention aims to provide a parallel processing method based on a TOP-K segmentation hypercube model. The method takes a hypercube model parallel processing model as a research object, improves the parallel processing capability based on the hypercube model, and improves the processing time, the expandability and the speed-up ratio of the method compared with the original method.
The technical scheme is as follows:
a parallel processing method based on a TOP-K segmentation hypercube model,
wherein, k: a natural number k;
TOP-K: the kth large value;
o (n): the time complexity is n;
d: the number of recursions;
O(log2 2p): time complexity log2 2p；
n: the number and size;
p: the number of parallel processing units;
the method comprises the following steps:
step 1: finding the kth large value;
step 1.1: selecting a reference value;
step 1.2: taking the reference value selected in the step 1.1 as a demarcation point, placing the number smaller than the reference value on the left side, and placing the number larger than the reference value on the right side;
step 1.3: and judging the position of the reference value and the size of k, if the position of the reference value is equal to k, indicating that a median is found, if the position of the reference value is greater than k, recursing the algorithm on the left side in the step 2 to perform recursion operation, and if the position of the reference value is less than k, recursing the algorithm on the right side in the step 1.2 to perform recursion operation.
Step 2: the value obtained in step 1 is the TOP-K value of the problem, which is taken as the central element. And broadcasts the hub to all nodes. And dividing the data into two parts according to the central element, and randomly distributing the two parts on the sub hypercube nodes. The time complexity of this stage is O (n).
And step 3: and (5) after the recursion step 1 and the recursion step 2 are carried out for d times, the original data are distributed on the nodes of the early-child hypercube. The processor interconnection interchange data blocks interconnected along the communication link of the (d- (i-1)) th dimension are subjected to calculation processing. Since at most there are
The elements need to be sent between nodes since broadcasting the center element at the ith iteration requires O (d- (i-1)) time. Thus, in all d ═ log2In p partitions, the time required for link communication is
And 4, step 4: and establishing a node interconnection link, and interconnecting and intercommunicating the nodes interconnected along the communication link of the nth dimension. The (n-1) -dimensional sub-hypercube with the nth bit of the node label being 0 contains the feature set in the original data, and the (n-1) -dimensional sub-hypercube with the nth bit of the node label being 1 contains the rest of the data in the original data. According to the programming goal, each node performs the calculation, and since p nodes are shared and the data is evenly distributed, the time complexity of the operation on the node is O ((n/p) log2p)。
And 5: each subcube recursively processes the owned subdata, and the data on each node are merged and summarized after being processed, so that a problem solving result is obtained. The combined solving time complexity of the segment with O (n/p) elements is log2(n/p), and finally, performing serial processing on each block of data, wherein the time complexity is O ((n/p) log2(n/p))。
Further, step 1.1 specifically comprises:
step 1.1.1: dividing n elements into
Each group comprises 5 elements, and if the rest elements are used as the filling number of 5 elements in the next division;
step 1.1.2: finding out the median of each group number in the step 1.1.1;
step 1.1.3: for all medians found in step 1.1.2 and the remainder in step 1.1.1, a new partition is made
Each group comprises 5 elements, and if the rest elements are used as the filling number of 5 elements in the next division;
step 1.1.4: and circularly calling the first three steps to obtain the median of the three steps as a reference value.
The invention has the beneficial effects that:
the parallel processing method based on the TOP-K segmentation hypercube model provided by the invention provides an optimal method based on the TOP-K segmentation, then improves the original parallel processing method based on the hypercube model according to the data segmentation mode, and further provides a parallel processing method based on the TOP-K segmentation hypercube model. The improved method has improved time complexity, expandability and speed-up ratio of the algorithm compared with the original method.
Drawings
FIG. 1 is a graph of the trend of the total time of d divisions;
FIG. 2 is a time complexity variation curve;
FIG. 3 is a scalability variation curve;
fig. 4 is an acceleration ratio change curve.
Detailed Description
The technical solution of the present invention will be described in further detail with reference to the following embodiments.
The effect of the invention is improved compared with the original method, which proves that:
suppose, in each of the d partitions, node P1The size of the sub data stored in the medium is increased by k times, and 1 <K 2. Thus, the total time spent in the d divisions is
When k > 1, the total time is O ((k)d-1) n/p). Since p is 2dThe above formula can be simplified into
When k is 2, then P1The time used for the division is O ((n-n/P), after d divisions, P1Has a sub-data size of 2dn/p。
When k is 1.1, the time taken for division is O ((p)0.138-1) n/p), the locally ordered sub-data size being n/p0.138。
When k is 1, the time taken for the division is O ((nlog)2p)/p), the sub-data size of the node is n/p, which is an ideal situation. It can be seen that the performance of the algorithm is worse when k is larger, as shown in table 1, and the trend of the total time change of the d-time division is shown in fig. 1.
Table 1: total time variation trend chart for d times of division
From the above trend of time-consuming total division, it is desirable that each time hypercube division is performed, sub-data of n/p size is allocated to each processing unit to be equally allocated, and such an allocation state is the best state. The total time taken for the segmentation is known from the previous derivation as O ((nlog)2p)/p) is also minimal.
Therefore, when the data is divided by taking the central element, if the selected central element is the median of the data segment, the data distributed to each processing unit is divided by hypercubeEvenly distributed, making the total time spent in the segmentation complex O ((nlog)2p)/p) is also minimal. The overall algorithm time complexity is reduced.
The invention provides a parallel processing method based on TOP-K segmentation hypercube model, the total algorithm execution time is
TpRefers to the total time a task runs on p nodes.
When (p log)2 2p)/(n*log2n)) ═ O (1), the solution is found to be p ═ O (n/log)2n)), therefore, when the number of nodes is equal to p, i.e., tasks are evenly distributed to each node, the distribution method is a cost-optimal strategy.
The method is proved to have improved parallel processing performance compared with the original hypercube.
Performance analysis and significant effects
The scalability scaleup and speed-up ratio speedup of parallel processing are two indexes for evaluating the quality of parallel processing, and scaleup refers to the performance of parallel processing after increasing the number of computers while enlarging the data size. scaleup (n), the time it takes to process a problem of size n using a single node running algorithm/the time it takes to process a problem of size n using n nodes running algorithms. If the scaleup value changes with n, if it is always around 1.0, or lower, it indicates that the algorithm has no effect on the change in size of the data set.
In the improved parallel processing method based on the TOP-K segmentation hypercube model, because
T when p approaches npIs composed of
scaleup(n)＝(nT)/TpThe value can be reduced to
Hypercube model-based parallel processing method before improvement
The analysis follows from the method after improvement of the acceleration ratio, the algorithm acceleration ratio speed (n) is the time spent processing the problem using a single node running algorithm/the time spent processing the problem using n node running algorithms.
Therefore, the first and second electrodes are formed on the substrate,
when p approaches n, it can be simplified to
The index shows the trend of the time taken for the method to run as the number of nodes increases. If a linear increase can be maintained, it means that the time required for multi-node parallel processing can be shortened well.
According to the analysis of the time complexity, the expandability scaleup and the speed-up ratio speeddup of the parallel processing method based on the TOP-K segmentation hypercube model, the performance estimation table of the method can be obtained as shown in the table 2.
Table 2: performance estimation table
From table 2, the trend of the time complexity of the improved method can be seen, as shown in fig. 2 below.
It can be seen from fig. 4 that, as the scale n changes, the time complexity of the improved parallel processing method based on the TOP-K segmentation hypercube model is lower than that of the method before the improvement.
From table 2, the scalability scaleup trend of the improved parallel processing method based on the TOP-K segmentation hypercube model is shown in fig. 3. The scaleup value of the improved method changes from the initial lower 0.8 to the vicinity of 1.0 with the change of n, and shows the trend of changing from low to high. The representation method has good adaptability to both the size change of the data set and the size change of the processing node.
As can be seen from Table 2, the speed-up ratio speedup trend of the improved parallel processing method based on the TOP-K segmentation hypercube model is shown in FIG. 4.
As can be seen from FIG. 4, the speed value of the improved parallel processing method based on the TOP-K segmentation hypercube model changes with the data scale n, and the time change trend spent on running the algorithm can keep an approximately linear increase from small to large, which means that the method can well shorten the required time when processing data with large scale.
The above description is only a preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, and any simple modifications or equivalent substitutions of the technical solutions that can be obviously obtained by those skilled in the art within the technical scope of the present invention are within the scope of the present invention.
Claims (1)
1. A parallel processing method based on TOP-K segmentation hypercube model is characterized in that,
k: a natural number k;
TOP-K: the kth large value;
o (n): the time complexity is n;
d: the number of recursions;
n: the number and size;
p: the number of parallel processing units;
the method comprises the following steps:
step 1: finding the kth large value;
step 1.1: selecting a reference value;
step 1.2: taking the reference value selected in the step 1.1 as a demarcation point, placing the number smaller than the reference value on the left side, and placing the number larger than the reference value on the right side;
step 1.3: judging the position of the reference value and the size of k, if the position of the reference value is equal to k, indicating that a median is found, if the position of the reference value is greater than k, performing left recursive operation in step 2, and if the position of the reference value is less than k, performing right recursive operation in step 1.2;
step 2: the value obtained in the step 1 is the TOP-K value of the problem, and the value is taken as a central element; broadcasting the center element to all nodes; dividing the data into two parts according to the center element, and randomly distributing the two parts on the sub hypercube nodes; the temporal complexity of this stage is o (n);
and step 3: after d recursion steps 1 and 2, distributing the original data to the nodes of the sub hypercube; performing calculation processing on the processor interconnection interchange data blocks which are interconnected along the communication link in the (d- (i-1)) th dimension; since at most there are
The element needs to be sent between nodes, because the broadcast center element needs O (d- (i-1)) time at the ith iteration; thus, in all d ═ log2In p partitions, the time required for link communication is
And 4, step 4: establishing a node interconnection link, and interconnecting and intercommunicating nodes interconnected along the nth-dimension communication link; dimension (n-1) with the nth bit of the node label being 0The sub hypercube contains a feature set in the original data, and the (n-1) -dimensional sub hypercube with the nth bit of the node label being 1 contains the rest data in the original data; according to the programming objective, each node performs calculation, and since p nodes are shared and data is distributed equally, the time complexity of the operation on the node is 0((n/p) log2p)；
And 5: each subcube recursively processes the owned subdata, and the data on each node are merged and summarized after being processed to obtain a problem solving result; the stage has O (n/p) elements for merging and solving, and the time complexity is log2(n/p), and finally, performing serial processing on each block of data, wherein the time complexity is O ((n/p) log2(n/p))；
The step 1.1 specifically comprises the following steps:
step 1.1.1: dividing n elements into
Each group comprises 5 elements, and if the rest elements are used as the filling number of 5 elements in the next division;
step 1.1.2: finding out the median of each group number in the step 1.1.1;
step 1.1.3: for all medians found in step 1.1.2 and the remainder in step 1.1.1, a new partition is made
Each group comprises 5 elements, and if the rest elements are used as the filling number of 5 elements in the next division;
step 1.1.4: and circularly calling the first three steps to obtain the median of the three steps as a reference value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810156673.4A CN108491187B (en) | 2018-02-11 | 2018-02-11 | Parallel processing method based on TOP-K segmentation hypercube model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810156673.4A CN108491187B (en) | 2018-02-11 | 2018-02-11 | Parallel processing method based on TOP-K segmentation hypercube model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108491187A CN108491187A (en) | 2018-09-04 |
| CN108491187B true CN108491187B (en) | 2021-06-11 |
Family
ID=63340621
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810156673.4A Active CN108491187B (en) | 2018-02-11 | 2018-02-11 | Parallel processing method based on TOP-K segmentation hypercube model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108491187B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN87106067A (en) * | 1986-08-29 | 1988-03-09 | 智能机器公司 | Super large scale computer |
| CN106790620A (en) * | 2016-12-30 | 2017-05-31 | 许昌学院 | A kind of distributed big data processing method |
-
2018
- 2018-02-11 CN CN201810156673.4A patent/CN108491187B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN87106067A (en) * | 1986-08-29 | 1988-03-09 | 智能机器公司 | Super large scale computer |
| CN106790620A (en) * | 2016-12-30 | 2017-05-31 | 许昌学院 | A kind of distributed big data processing method |
Non-Patent Citations (3)
| Title |
|---|
| BFPRT (中位数的中位数）算法;BENULL;《CSDN博客 https://blog.csdn.net/qq_32815807/article/details/78183597》;20171009;全文 * |
| Fast median search: an ANSI C implementation;Nicolas Devillard;《百度学术》;19980731;全文 * |
| 超立方体计算机上的最近点对算法;杨波，庄心谷;《西安电子科技大学学报》;19951230(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108491187A (en) | 2018-09-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9298760B1 (en) | Method for shard assignment in a large-scale data processing job | |
| US11055139B2 (en) | Smart accelerator allocation and reclamation for deep learning jobs in a computing cluster | |
| CN112015366B (en) | Data sorting method, data sorting device and database system | |
| US11221890B2 (en) | Systems and methods for dynamic partitioning in distributed environments | |
| WO2017118335A1 (en) | Mapping method and device | |
| US10162830B2 (en) | Systems and methods for dynamic partitioning in distributed environments | |
| CN112235344A (en) | Distributed machine learning-oriented sparse communication model implementation method | |
| CN112085644A (en) | Multi-column data sorting method and device, readable storage medium and electronic equipment | |
| CA3094727C (en) | Transaction processing method and system, and server | |
| Durad et al. | Performance analysis of parallel sorting algorithms using MPI | |
| CN108491187B (en) | Parallel processing method based on TOP-K segmentation hypercube model | |
| CN116701001A (en) | Target task allocation method and device, electronic equipment and storage medium | |
| CN112232401A (en) | Data classification method based on differential privacy and random gradient descent | |
| CN109254844B (en) | Triangle calculation method of large-scale graph | |
| CN115705497B (en) | Quantum computer operating system and quantum computer | |
| CN111143456B (en) | Spark-based Cassandra data import method, device, equipment and medium | |
| US11442792B2 (en) | Systems and methods for dynamic partitioning in distributed environments | |
| CN111737347B (en) | Method and device for sequentially segmenting data on Spark platform | |
| Gao et al. | FT-INDEX: A distributed indexing scheme for switch-centric cloud storage system | |
| US20210405904A1 (en) | Method, device, and computer program product for managing storage system | |
| CN106997303B (en) | MapReduce-based big data approximate processing method | |
| CN114819195A (en) | Training method, device and system of ensemble learning model and related equipment | |
| VanCompernolle et al. | Maximum clique solver using bitsets on GPUs | |
| Cinel et al. | A distributed heuristic algorithm for the rectilinear steiner minimal tree problem | |
| CN117112145B (en) | Training model distribution method, training model distribution device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231123 Address after: Room 1010, Apartment Office Building, No.14 Xingqing Huafu, Xingqing District, Yinchuan City, Ningxia Hui Autonomous Region, 750000 Patentee after: Ningxia Yifan Weining Technology Co.,Ltd. Address before: No. n817, 3rd floor, xingguangyingjing, No. 117, Shuiyin Road, Yuexiu District, Guangzhou, Guangdong 510000 Patentee before: Zhiyueyun (Guangzhou) Digital Information Technology Co.,Ltd. Effective date of registration: 20231123 Address after: No. n817, 3rd floor, xingguangyingjing, No. 117, Shuiyin Road, Yuexiu District, Guangzhou, Guangdong 510000 Patentee after: Zhiyueyun (Guangzhou) Digital Information Technology Co.,Ltd. Address before: 461000 No. 88 Bayi Road, Henan, Xuchang Patentee before: XUCHANG University |